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/ INTRODUCTION ' 

Administrators ^re surely very busy people. There are ever more tasks . 
to attend to, eVer more cc:*^';erns and problems that require time, energy, 
and concentration. Yet here you ar« with a book about evaluation in^our 
Ji(yids,,J^ guess, then, that one of your concerns is likely to be an evalua- 
tion of a project w,ith which you are associated, or for which you have 
administrative responsibility. In view of this concern, there may be a 
variety of reasons motivating you to examine this book. 

Perhaps evaluation is relatively new to you,' and you would like some 
clarification about your role in- the process. Or perhaps you are wondering 
how worthwhile expenditures of time and resources^ for the evaluation are 

— likely -i:o"~fae-and - would -l ike to hel p i 

worth." - \ 



Or, you may already have experience participating in evaluation, V|HJt 



been disappointed in the outcome. This -time , you'd like to take an act/ive 
role in making si>ni5^hat the evalijatioTrn-s betterH:arh)red to your- project Vs 
needs , "though you ~ma7^ t^^^^^^ 



We'd like to reassure you — you have come to the right place to get 
immediate help. The^uthors of this volume provide the fbl lowing assurances: 

• This handbook is oriented spetiflcally to those who commission or 

are ^otherwise administratively responsible for seeing that evaluations 
are 'performed. ^ 

• This handbook is practical; we don't just tell you ^^^^^w^^^^ to do. Rather, 
^ we show you how to do it and provide you worksheets for practice. 
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• This handbook is easy to read, vyith a minimum of technical . language 
and a maximum of conimonsense 'advice. -v/ 

• This handbook is useful for administratttrs at any stage of an 
evaluation.,- whether l-n th^ planning process or nsaring completion. 

Evaluation 



Let's clarify for a^ moment -what "we mean by "evaluation." When we lise the 
term in this handbook, we are referriJig to the activity of systematically 
obtaining information that might be used to change attitudes or to make 
judgements which can be used to improve the operation of projects. What 
does this definition mean in practice? 

First, note that the word "systematic" implies that there must be a 
plan to the evaluation. Evaluation is not simply the acquisition of data 

-or information in some haphazard fashion. This organized plan for data (or 

i rif orma ti on ) colT ec ti on s hou^l d be di rec ted -toward obtai n i ng: da ta iha t_wi 1 1 _ 

provide answers specifically to the questions or concerns you or others may 
have about the project. You need to consider carefully how the evaluation 
plan will help obtain that data. After Ideterraining the questions and concerns 
"obtaining the desired information" involves several activities: identifying 

■^Jthe„appropri a te i n f ormati on , acqui ring the information, analysing the data, 
and forming justifiable conclusions based on t|e data^ 

^ Once the appropriate information h§K,^n obtained and analyzed, then 
the conclusions based on this data can be put to use in two major ways. ^ 

' The first of these, stated in our definition, refers to the changing of an. 
attitude or belief about the project or its operation. That is, the findings 
from the evaluation might influence your (or sonieone else^s ) 'opinion about 
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the project or the way it operates^. (While -this might have no immediate • 
visible impact on the project^ it very well could change or influence actions 
in;the future.) A second way that" you could gse the^ conclusions based on 

■: . . O.s . ' ' 

the data woQld be^to make actual decisions relating Jto project operations or 
even to' project continuation. For example, ba^ed on the -evaluation findings, 
, you might recornnend that the project not be refunded forVthe following year, 
or els^ you might make certain modifications in the way'^that the project 

operates. , . , ' ^v' v;: ^ 

>3 • . . . 1 

■ . , . I ^ \ 

O ' - ■ V • . . ■ ; 

But there's another important f eature^ of this definition: we use 

^^the term "evaluation" to refer bnlyvto evaluation, "of an organizational 
" ^ ■ " .... - \ 

entity (such as a "project" or program). We are not discussing here the 

0 evaluation of individuals (such as students, trainees-, personnel, managers, 
etc.). We are discussing projects that operate in organizations such as 

~schpols,-social -service agencies, hospital etc- These project^ 

can be at a micro level within the organizat^/on (for example, the thirds- 
grade mathematics program in one ejementary school), or they may be' at the / 
macro level (for example, the Aid to Families with Dependent Children. s^bciaT ' 
services statewide)/ In both these examples, though the data collected 
come from individual participants in the project (either the third-grade f V*" 
students or the AFDC clients), the evaluation- deals with the entire pro jec^^^ 
as the entity to be judged, and not with the individuals who have provided 
the. "data for analysis. . ~ • 
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Evaluation as a Meaningful Activity 



\ 



Our firm belief, and one that we will endeavor to transmit' to you 
?^lhroughout this handbook, is that evaluations should be meaninqfut to all 

those involved in the process. Evaluations shoiild have importance, signi- 
„ficance, and purpose. In short, the evaluations must be more than window 
dressing"— they must fulfill, a necessary function. We^re sure that many 
of you reading this, handbook will recall instances, of evaluations — not 
meaningful — which produced nothing more than a dull report to be forever v 
filed away on a dusty shelf. No morel j . ^ . " 

So what does "meaningful" entail? The first issue is, meaningful to 

■ ■ " / ' ^ • ■ ■ 

whom? To statisticians, to researchers, to journal' publishers? ~ Nol 

Meaningful to funding agencies, tp f^igher-l^vel administrators, to the 

' / ■ / / 
evaluator? — Maybel But primarily the evaluation must be meaningful, 

significant, and importarit to you and to' .others who are actively involved 

with the project and are making decisions about it. 

to further understand "meaningful ," you need only think about what ' - - 
yoli would personally consider meaningful. We believe you would think an 
evaluation meaningful if it were relevant*, credible,' and>actively involving 
)jou in its process. ' _ ' ^ , 

A relevant evaluation s^hould be "on target." It must deal with topical/ . 
critical issues (issues of concern to you and to yoyr colleagues), and it^ 
must deal with them in an appropriate fashion. 

A meaningful evaluation should also be credible. There are several 
Important aspects to credibility. .First, the evaluation should have technical 
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credibility (i .e. ^ 'statisticaVcorrectness, properly crafted Instrum^ts, 
valid design, etc.). The technical credibility of the evaluation should be. 
at a sufficiently' high level that you and others can feel that the results - \ 
have been appropriately obtained, a/nd are beyond major professional criticism. 
We note parenthetically that almost every report can be critiqued and found 
^ to contain, some technical deficiencies: what concerns us here is avoiding 
glaring inadequacies. 

But there's another side to crediBl.lW — let's call it professional 
credibility. Quite simply, ask yourself iflthe evaluator is someone you 
can believe in—- and, by implication, is the evaluation itself something 
you can tru^t? Do the evaluator's actionr reports, .rjBcommendatipns instili 
sufficient confidence thdit you might want to go ahead, to mpve on tfiem, to 
make decisions, to make changes where necessary? 



A further dimension of "meaningful" evaluation is that of your own 
involvement in the evaluation. Ask yourself how drawn you feel to getting 
involved, to wanting to be close to what is happening. If the eva^-uation is 
but a minor consideration among other, more pressing priorities, you ars • ^ 
unlikely to make a majoi* c^onmitment to it. And without this close involvement^ 
'on your part, the evaluation; un^^oubtedly. will lose ^credibility. • 

In sum, then, for an evaluatiqn to be meaningful , you must be able to 
believe in the evaluation's purpose, to. trust its processes and findings, 
iand to participate actively in its^conduct". Such active involveffient will 



not only help make the^evaluation meaningful to yog,' but by your example will/ 
also encourage meaningful .involvement of 'others close to- the project. 



, Evaluation as an Organized MeanirigfuT Activity . . ' , 

/ ... ' ' , * ' ■ 

y Meaningful evaluations -as we have just described them do not come about 
by chance ~ they must be planned for, carefully jorganized, properly carried 
out, and effectively conimunicated to people who intend tia use that informaition. 
And this is where you can make your greatest. contribution. ^ 

We consider that* you*, in your administrative role, 'are the ^"organizer" 
of the evaluation. In terms of our definition of "organizer," this makes „ 
you the person ultimately responsible for the evaluation. ' You play the most 
important part of all in -assuring that the evaluation will turn out to be 
. meaningful. Time and time again., research has shown that it is the presence 
of a person in a position, of authority who takes a personal interest in the 
evaluation that makes the difference in whether or ,not ^esQlts of the 



" e va 1 ua tTdri" a re ac t uYT l y pu 1 1 to u s e~ 



• - and that i s c erta in Ty a prime fhdi ca- 



tion that a meaningful evaluation h^is occurred. ' \ ^ - V 

V A "But," you may be saying, "I aln only one, person. How can I be so- 

' f-' ^ - : ; ■ „■ c -'. .... ■ ; o ^ 

important? And besides^ I have lots of- other responsibilities to attend 



to tother than this 'one evaluation.- 
everything yourself 1 



" We are not saying that you must do 



the key to successfully carry^'ng oiit J^our role is understanding how/ 
to organize, how to structure the evaluation process so that the other/ ' 
sMnterested parties in the evaluation can be involved to best advantage. 
And you are the one person familiar enough with the context in whic^ the 
evalutibn will takfe* place to organize and involve everyone 'Connecj^d with 
the evaluation process — yourself, as administrator; the evalw^^ the 
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project staff; the project participants; and any other interested parties, 

whether internal or external tp^your own organization;' . ' » ' 

■, ■ , ' ' ' ■ . ' ■ • ■ . 

■ . ' i:" ; ' ' . ■ • . •'. , ■ ■ ' ..^ ^ 

• This handbook jWi 11 give you the infor.*iation (and the tips) you'll lieed 
in your administrative role as the evaluation '^organizer" to manage the best, 
most useful evaluation possible within the context of your project's realities< 
. It will take you through the-early stages of the evaluation, which we've 
termed "Setting the sta^e" and 'nidentifying/organi zing the participants"; 
^ through the executio^ of the evaluation activities ("Operationalizing the 

interactive process"); and finally through the concTt^di/ng phase of the 
\ evaluation ("Adding the finishing touches for use"). Here's a brief sample. 



drawri f rom the stage labeled "Identifying/organizing the participants'' : 

■'y .. ' / '-^ ' \ ■ ' 

Consider for a moment the evaluator. Perhaps you already .kno)y some /' 
^ evaluators; there are lots of them available nowadays. Some evaluators 
may act like remote, clinical researchers who want you to set up 
controlled experiments in order to, evaluate your .project. Some 
evaluators may tell you what needs 'to be,evaluated, with or without 
your or other interested parties' concerns and questions about the 
. project being 'discussed and. taken into consideration. Still other 
evaluators may offer you little/or no professional guidance in design- . 
ing the evaluation. If you've worked with any of these kinds of .' I 
^ evaluators before, we offer our condolences, and our encouragement 

that this time ,\ybur evaluation should have an evaluator it merits, 
an evaluator who is oriented toward the users' concerns and toward - 
producing a meaningful evaluation, a useful evaluation for decision- 
making. \ \ \-. rv. o ' 

- ■ A ■ :.■ ■ ' ' - ■'>■•' 

What does such an e^luator Iook like in action? -THe^evaluator you 
^ should be seeking to\h1re should ideally be' someone who^W'^nts to inv 
"^ you and the other potential users of evaluatign information in the 
planning and the conduct. of the evaluation; wfip asks for^^^^y^^^^ 
and concerns about the'^project to be eval uatedV who has good interpe^ 
sonal communication skills; who can assess the "potential 'users' . pre- 
ferences for reQeiving infonnation and responds 'appro 
who is politic^ly aware... A^ • ' \ 

We hope the above example has provided spme hint that this handbook 
does have relevant information to offer you, Ancl if=we haven't yet per- 

o ■ ' ■ ■ ■ ■ ' It ' ' * ' ■ " ' ■ 
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"suaded you r tHen perKa^ we Have at' least caught your irrterest stiff iGientTy 
that you wilt continue reading before making up your 'mind. 

We conclude this brief introduction by outlining^ the- content of the 



remainder of the handbook. Chapter ni 'disc^^^^^ the idea of evaluation use, 

■ ■ , »> . 

one. of the central concepts injproject evaluation. It also introduces 
the notioa of "Factor Patterns," cliisters df individual elements which can 
help you organize your evaluation into recognizable, defined stages that 
promote evaluation use. Numerous examples accompany the definitions and 
discussions. ^ • 

Chapter III goes on to discuss in detail each of the individual clusters 
of elements which ma^e up the FActor Patterns. You will find suggestions 

of the elements ..work for you as you. organize the evalua- 



/ 



for how to m^ke each 

tion. You'll aJso ^^*nd.tacti\:s and strategies you* xan employ if you identify 
elements, in ^your own evaluation situation which could potentially work against 
evaluation use. 

Chapter ylV provides a detailed scenario of one evaluation context. 
Seen through the eyes of the responsible "organizer, ".J:his scenario provides 

■ r ■ ■ \ ■ /'■■^v/, ■ ■ :;■ ^ .-, 

the background against which the organizer shares her thoughts and her 



\activities as the evaluation progresses from the planning stagis, through the 
operational st4ge, and finally the \c1b5ing stage. Jhe discussion charac- 
terrzes the evaluation in terms of the four Factor Patterns introduced in 
ChapterMI and III, and focuses specifically on the issue of maximizing (he 
potentiainfervevaluatipn/use* 



Chapter V completes this handbook by providing worksheets for you to: 
use (or practice with) as you consider organizing your own evaluation. By 
the ime you finish this chapter, you will become quite expert as an 
evaluation organizer I 

We know that you are anxious to get started, so we suggest that 
without/further delay, you turn to Chapter II and begin reading about 



CHAPTER II 
WHAT IS EVALUATION USE? 



This chapter deals with a central issue in project evaluation 
the use of evaluation as a decisionmaking and administrative tool 
Administrators in charge of projects ~ people like yourself — want to 
run their projects as efficiently as possible and. make sure that the 
projects are as effective as possible. Evaluation can help in these 
tasks, but only if the information produced by the evaluation is received 
and used by those who seed it most. Our intention in this book-isto show- 



you how to organize an evaluation for the express purpose of piaximizing 
its potential for use. 



Def i ni ti op^of^ey-^Terms 
-^__:^__Welve-^just^*ht'foduced a few key terms into our discussion, terms 
which should be clarified before 4) roceeding Vurther. The term project' 
refers to an organized undertaking. Such an undertaking generally requ\ires 
a staff whose members have different jobs and responsibilities related ^0 : 
carrying out the goals, objectives, and activities of the program being ^ 
implemented. Here are a few examples of what we mean by "project": 



A school district introduces bilingual instructional pro- 
cedures to teach English and other subjects to limited 
English-speaking students. 



A medical center introduces training to improve nursing 
educatorsV abilities to develop nursing competency tests, * 

A bank^'s main off ice^devel ops prbcedures for showing branch 
managers I hbw to implement updated sol^tware in thei r compu- 
terized data-proce'siing system; 

• An i hf antry di vi s^on • s commaftder i nsti tutes f i el d-trai ni ng 
exercises to train company-grade officers in long-range 
reconnaissance tactics* * 
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A corporation's sales division plans a conference 
to introduce the successful promotiohal and sales 
strategies of one region into other regions. 



Each of these projects could be subject to evaluation. 



An evaluation collects, analyzes, and* reports information on some 
undertaking such as the projects listed above. It is designed to produce 
information that will help in decisionmaking: for example, decisions 
concerned with improving the efficiency of project operation. . 




Here are some evaluation examples drawn from the five projects 
identified above: 



The school district's bilingual project 'staff needs 
evaluation Information to make sure that students who^ 
need bilingual services are being prope^J^ tdentlfled. 
They also need information on how well the^instructlonal 
component is being Implemented, on how successful the 
project Is In preparing students for Instruction in 
English, and on how well the Individual schools In the 
district are doing. this job. j 

The medical center's training unit needs evaluation 
information on nursing educators.' attitudes toward 
testing. They also need to know how successful the 
training program Is in teaching /nursing educators about 
test development. That Is, are /the tests they develop 
after receiving training of higher quality than the tests 
they developed prior to tra1nir)'g. 

The senior data^processlng staff of the bank^s main office 

J»ultl^bsneflt_:^fro 

attitudes toward the proposed /changes In ithe software system, 
on the extent to which locaUjbranches have a^s^^ 
system pri or to the ^1 ntroduct/ioh of changes v and on ways /to 
Improve local operations. 
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The 1 nfantry di visi on commarider and his headquarters staiff 
need Information on the current asslgnmehts^o 
1 ong- range patrol train 1 ng / They especi al iy ^ need 1 ocat] on 
and status information about combat veterans with long-/ 
range patrol .experience wha ire avail abTe for te'mporarW 
duty, to thevtraining effort. During .tralhing, the staiff 
needs information collected on the ground. - If the training 
lis di vlsionwl^; and^lnt^^^ to nieet contingency needs / only • 



division staff will need information on the general^ 
performance of company-grade officers in different ^ 
battle groups If the training is intended to screen 
and select commanders of long-range units, the staff 
will need specific information on each candidate's 
performance. 

The corporation's senior sales executives need infor- 
mation on how well the conference was received by the 
participants. They may also want follow-up, information 
on whether the successful promotion and sales strategies 
were implemented as intended; Finally, they need infor- 
mation on whether regional sales volumes remained the 
same, increased, vir decreased after the proposed strategies 
•took, effect. l_.l__-l_:__„^._ _ 



As these examples show, an evaluation can collect different kinds of 
information at different points in time: before a project begins, 
during the course of the project, and at the^ end of the project. More- 
over, different kinds of personnel can be involved both in the project 
itself and in its evaluation. 

project may also have several administrators working at different 
levels in the overall organization. In this book we will focus primarily 
on the particular administrator with operational authority oyer and 
. responsibility for the project being evaluated. We recognize thgit this 
administrator may be designated by a. variety of titles: project director, 
principal/, program manager, project supervisor, etc. Whatever his or her 
ti tle* this individual , in essence, performs the functions described below ^ 
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The project administrator is the person responsible for the. overall 
operation of the project, the person whose actions and influence usually^ 
reach into every aspect' of the project and whose administrative responsibilities 
include dealing with/every group or agency with an interest in the project 
and a stake In its operational procedures "and outcomes. One important < 
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function of this administrator is to work closely with the evaliiator 
throughout the entire evaluation process. 

The following examples, drawn from the cases previously described, 
identify the person whomay best be regarded as the "project aaministrator," 
as defined above: ^ 





In the school district, the director of bilingual ^ 
education is likely to be the project administrator. 
Although he/she may report to others with some influence 
over the project at the broad policy level (e.g., an 
assistant superintendent in charge of compensatory 
education, the superintendent of the school district) 
nonetheless it is the alrector of bilingual education who 
is responsible for the operation of the bilingual project 
and for its evaluation. 

In the medical center, the project adrhi hi strator is - 
probably the director of nursing education. This person 
may work with and report to other administrators. But 
by virtue of position and training, s/he is best equipped 
to identify area? for which evaluation, information is needed. 

^In the case of the bank, the project administrator Is likely 
to be some fairly senior Individual in charge of system- • 
wide computerized data processing. This Individual could 
be a vice president who is knowledgeable about both banking 
operations and data processing and thus is therefore stra- 
teyicai ly piaceu to consiqer evaiuatiOn Within the total 
sphere of banking operations, not only computer systems. 

In the Infantry division, there are at least two possible 
candidates for project administrator. In the case of 
divlsionwide contingency training, the division commander 
is ultimately responsible; he may, however, assign an 


• 




assi stant as project ai rector . In the case of uni t-commander 
screening andiselection, the designated project director 
might be a battalion executive officer with appropriate^^- 
training and experience. - 

In the, corporate sal es divi sion ^ the project administrator 
will probably be a senior executive in-'the headquarters' 
sales diylsiori. While this person reports to the national ' 
head of Sales', s/he assumes direct responsibility for .running 
the sales project and i-ts evaluation. - 
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Each of the project administrators identified above, then, would 
have operational responsibility for the project and. its evaluation. We 
contend that this responsibility entails the promotion of evaluation use. 

I Evaluation Use . 

What exactly do we mean by "evaluation use"? The term refers to the 
purposeful, planned consequences that result from applying evaluation 
/Information to a problem, question,. or cohqern at hand. Another of ' 
phrasing the definition Is to say that "liseXis the application of evalu- 
ation information or evaluation processes to achieve intended, desired ends. 

Two questions arise naturally from the above definition: use by 
whom? and for whSt purpose (s)? Let's deal with the first question how. 

Use by Whom? 

. • *^ " • ■ , - ■ ■ ■ ■ ■ • 

There are many potentlaT "users" of evaluation information. For 
example, when you look at ^ifhe program to be evaluated (e.g., the program 
for which you have, overal 1 resporis1b1l1ty) , consider who the persons are 
that actual ly-make the program work. Perhaps you have a staff , a desig- 
nated project director, or several subadm1n1strator$. Perhaps your project 
invol ves-xli^irts-^^ 

groups outside youn'own organization. Each of these groups can have a 
• professional and a. personal interest 1r| the information to be produced by 
the evaluation; each is a potential user of evaluation. In addition, many 
projects have another outs ide audi encel that can be a potehti al eval uation 
user -- a state or federal ag^ency, which often mandate that^an evaluation be 
performed as a c6nd1 11 on of gran^i ng/f unds for pro j ect operation . 



Use for What Purpose (s )? 

The ranqe of possible evaluation users is large, but there are only 
three main categories of evaTugiioQ use. The first of these is termed 
"action use": the intenttdnal. use of evaluation to inform decisions 
about the. project being evaluated. An obvious example of action use is a 
changie, made on the basis of evaluation information, in a project's content 
or methods. Another example: of action use is avconclusicn^ 
basis of^evaluation information that the program is running as intended 
and does not, need to be modified at the present time. ... T 

Use may result from an evaluation's processes as well as from the 
data it produces. For example, contacts between the ^evaluator and the 
project staff early in the life of the project may result in shared concerns 
- about the need for project modifications, which in turn may lead to actual 
modifications in project operation. Or the evaluator may Indicate that 
certain data will be collected as part of the evaluation. An understanding 
of the measures to be used may lead to a recognition of project deficiencies 
(and changes) before any evaluation data has been collected. 

The second kind of use is conceptual in nature. The tenn "conceptual 
use" refers to the intentional use of evaluation to modify people's atti- \ 
tudes toward the project. Often,- conceptual use arises from the personal ^ 
exchanges and Interactions that take place between the evaluator and : 
potential users, or between the e^luator and the project staff. By its 
nature, ^conceptual use does not i/ivo1ve any tangible or identifiable action. 
Nor can one point to any written document to prove that conceptual us el has 
occurred. Rather, conceptual use is manifested by changes in attitude- and, 
where possible,-- in siibsequent behavior. . ' . • :. 



Thiird, the professibnal literature talks about the "symbolic" use 

/ - ■ 

of evaluation, as when an institution /''joes through the motions" to show 
that something required is taking place, that regulations are being 
followed./ Such symbol ic -use is seen, for example, when an institution 
contract^ to have an evaluation conducted merely to demonstrate compliance 
for an ©eternal agency. The evaluation itself is a way of saying, symboli- 
cally, "Look, we are complying with the regulations by submitting our^ 
project to evaluation." . 

/ • - ^ ■■■ .. . Q . . . • ■ 

' Throughout this Guide we will be concerned only with the first two 
/ uses of evaluation: ^caction and conceptual . We will not deal with symboTi 
use for two reasons. First, symbolic use is intiirent in the conduct of 
the evaluation itself. Second, we do not want to encourage evaluations 
directed only toward symbolic purposes. Such evaluations are a waste of 
money and time. This book is directed toward making evaluations truly 
useful so that their conduct will be worthwhile. 

Likewise, an evaluation can have both intended and unintended uses. 
Occasionally, the results of an evaluat'ion. can be used in ways that are 
very different from what. was envisioned. For example, the use of the 
evaluation report by a coirenunity organizaiticri to demonstrate lack of 
sufficient support from other agencies. Sometimes these unintended useS^ 
are beneficial i sometimes not. We might refer to this as 'the "Impact" of 
the evaluation in contrast to Its "use," which is planned and intended. 
Since this book is focused on getting you to anticipate and improve the 
use of evaluation, we win not consider Impact In this broader sense. 



^Here are a few observations which you should keep in mind when you 
consider "organizing ypu^ evaluation for action use, for conceptual use, 
or for both: 



i 



0 First, not every intended acti-on is preceded by I 
conceptual use. But,;Where action d&pends on.,=, 
prior conceptual groundwork, yoii shoufd- be^siilre 
that this groundwork is 'laid. ° ^ , 

0 Second, and conversely, not every conceptual use ' 
. is fp.llowed cby an action. But where- such progression 
is warranted, you should make sure that vjX bccurs. 

0 Third, conceptual use, °jn itself, may be tfje final 
des;ired effect of many evaluations. ^ / 



Process and Product 



As indicated in the previous discussion, evaluati^Sn/use "can be 

stimulated by the process of evaluation (such as data/collection), byo 

the product of the evaluation (such as a report); or/by some cornbination 
■ : - ■ ■ ■ : :// -'. ■• ■ 

^of- the two. Let's define these terms a bit more narrowly. 

; ' ■ -"v- ' / ■■-■■ : \ .. 

By evaluation process , we mean the ways in whj'ch the evaluation 
is conducted.- for example, the involvement of selected \individuals as 
informants or" interviewees; the formal and infornjal interactions between 
the eval uator and_the users during the s tudy ; t>i4 data coll prf^nn p rnrft>^^^rp'; 

■ ^ . X / •7-:^^.-, 

The evaluation process can-affect a project simply by virtue of the 
.fact that itfisr^)ccurring and is thereby involving' project personnel . For 
exampl e , contacts between -the eval ua tor and /the staff are a natural part 
of the use-oriented eWluationaprocess. Si|;ch contacts, may lead to" changes 
in the attitudes, behavioH or actions of ^staff members. To take another 



example, the project director, in supervising data collection for the 
evaluation, may develop a closer working relationship with the project 
staff. This enhanced rapport results from the evaluation process itself, 
whatever the ultimate findings of the evaluation. ■ 



evaluation product , we mean the organized body of findings, data, 
recommendations, and other information provided at appropriate points 
during the evaluation. The evaluation product includes, of course, the 
final evaluation report, but it may also include interim reports, findings, 
and recoimendations. 



The evaluation product caif affect a project fhrough people' s expecta- 
tions of the findings. For example, a teacher may change course content 
or teaching methodology on the'basis.of preliminary evaluation findings 
about student achievement. /AsTanother ^example of the effect of evaluation 
products, a citi zen's advisory group may demand' that certain "project 
changes be made after hearing some of the evaluator's recommendations. 



Let's^QQ back now" to the^ case examples introduced earlier and 

■ « ■ h . ■ - . ■" ■ 

examine theVange of possible users and uses. 



The school district bilingual director may be concerned about 
~^Ke-neg^tlve^Atyj:ud^^^ teachers (i .e. , those outside 

the bilingual project) ^oward^the pxojec^ outside 
teachers constitute one possible group^f usersv— The-d1rector^_- „ 
Intends that favorable evaluation Information Instil 1 more 
positive teacher attitudes toward bilingual* education (concepr ' 
tual use). The director may believe that the evaluation v 
indicates a need to convince the sup6rintendent and the school 
board (another group of potenti al users ) that the bi 1 ingual 
/program should be expanded. Such an expansion would constitute 
/ an example of action use, preceded by conceptual use (convincing 
the board that expansion is 'necessary). 



The director of nursing education may find that the evalua~~ 
tion results show' that the st^a ft Tacks the -skins necessary 
for developing' sound, valid competency tests- The staff , 
then,, is a potential user of evaluation. The director, 
decides to -institute- some staff training (an action use). 
The director also uses the evaluation findings to make • ' 
recorrenendations to the board of directors (another group 
of potential users) about updafting nursing education ' 
courses. " This constitutes an example of conceptual use. • \ 
The actual updating of th6' courses would constitute action 
use. ■ ' --r: \' 'vr^ ^\ ''^ 

The bank's project, administratorj may discover through the ^ 
evalugition that sorns branch managers- have not kept, up with 
recent 'technological advances so l that .they cannot take ful l 
advantage of the/training program. This administrator, . 
''using the evaluation data, design a brief'Irefresher course 
for these managers (action ,use). 1 In addition,- the evalua- 
tion results identify managers who/appear to be extremely | 
khpwleiigeable and competent. ,Theiadministrator yses this ; - 
information to come up with some policy recommendations 
for' staff promotion and advancement (ccnceptual use), r 

The infantry division project director, after examining 
the evaluation re.sultsr may find that not all battVe groups / 
in his division are equally combat-ready. He uses th|. 
information'ta tir^ift; recommendations to his* superiors; for. 
further training (iconceptual use). He further uses the f 
evaluation information to make addit\lpnal recomnendations . 
regarding the role of senior NCOs in the training process. f; 
In this instance, conceptual use of evaluation was the /V • 
intended, desired use. to the extentl that his recoinnendations-^ 
are implemented, the evaluation also results in action use.\ 

The corporate sales project administrator intends that the ' 
evaluation data be, used to design subsequent training for ^ 
s/taff (conceptual use) . However » when the data are actual ly . 
received, an' iniiiediate need for additional^^^^^i region 
/Oecb mes apparent. The^administr ator t^^^ to r 

'^transfer staff (an action use, rthoiigh; previously unplanned)-. 



Orqanlzing for Evaluation Use:: An Overview 



We've now discussed Several important:, facets of ^w^^ 



ti/led as af major issue in' project evaluation: : nam^^^^ 



/ 



Our discussion next leads us to the question of organizing for evaluation use 



We already know that, withiVany given evaluation, there is generally 

V '^^ \ . . 

one individual who fulfills^the funb^aoii^of a project administrator; that, 
there are two possible uses action lind conceptual toward which the 
evaluation is aimed; apd that there are a- multitude of ^possible users of 
' evaluation information:^ And we've already shared our belief that a major 
responsibility! of the persor^^den^ as project administrator, is to 
promote the us'e of evaluation.V So what •can the administrator do to organize 
the evaluation in such a manner that its, potential for use is enhanced? 

.'*■■■ ; . ■ \- ■ ■ ■ • ■ ■ 

'The key to understanding the ro^l^ of the administrator in "structuring 
the evaluation process .is to appreciate. the\compl ex array of circumstances 
that surround an evaluation. Within each evaluatiQn situation there are a 

.multiplicity of factors which can have an effect on evaluation use. A 
f actor ^ then, is^any characteristic or element present in a. given situation 

; which can affect the outcome of that situation. When ain^ administrator is 
cognizant of th^'se factors, an^ when he or she: actively structures the 
evaluatto^n to Hake maximum' advantage of those factors known to favor evalua- 
tion' use, then the. poteatiaT for evaluation use is enhanced. 

■ Let;',s Jook a bit more clds^^^^ which can be 

.divided into, three' categories: human factors, context factors , and evaluation 
factors, 

The first category .'^ human factors , ^includes, the characteristics^ of / 
the eyaTuator,c.of- the pcople^connected with the project;, and of any other • 
\ potential users«^ , ^For example,^ the evaluator,, as a human being,^^^^ an 
Indi vi du^il set of personal character! sti cs , 1 i kes', and disl i kes . Each of ^ 
these chVacteristic^^can affect"" the evaluator's dealings with the other 



human beings involved in the evalxiation. Similarly, the persoWl charac- 
teristics and preferences of the users interact with those of the -evaluator 
within the evaluation setting. Among the many human factors that can 'o ^ 
influence evaluation use'are people's attitudes toward "feyaluation,, toward 
the evaluator, and toward the project; and the evaluator's rapport with ° 
the various user groups. - 



The second category, context factors , reflects the specific context " 
in which the evaluation is conducted. Included here are such elemeVits as - 
fiscal constraints oa the evaluation, the length of program operation^' apd 
the social/political 'climate "surrounding; the project. ^_ ■ ' 

The third category, evaluation factors , involves the evaluation itself. 
Included here are such elements as the evaluation procedures used, the 
kinds of information collected, and the way in which the information is 
reported. 

We can illustrate each of t^e three categories of factors from our 



case-study examples, 



In the s c hool district, the project director identifies 
specific issues f^r the evaluation to address. On the 
basis of this input, the evaluator designs prbcedures 
to gather the data needed to address these issues. This 
factor, termed "evaluation procedures,"' belongs to the 
third group of factors. 

In the medical center^ the director of nursing education 
senses a defensive attitude on the part of the nursing 
educators; with res jsect to the forthcoming evaluation. 
This attitude reflects the factor termed "a ttitucie ■ 
toward evaluation", which belongs to the\first group. 
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^ In the bank, the limited fiscal resources for the evalua- 
' tion lead the project administrator to concentrate on the 
information needs of the, branch managers to the exclusjl^on 
of other staff / . This targeting of funds and energy | 
reflects a contextual fc^ctor*. "fiscal constraints". | 






In. the infantry division, the project director recognizes 
that battle group commanders need to have a fair amount 
of tactical authority in or^der to use the evaluatton 
information. This factor, "unit-level autonomy," belongs 
to the second category. 

In the corporation, the project admniitrator tells the 
evaluator that the chief executive officer prefers reports 
' that have a lot of graphics but relatively little text. 
Two factors are involyed here: "user information-pro- 
cessing preferences," from the first category; and "format 
of report," from the third category! 





CHapter III of this Guide Includes a full description of all the 
individual factors in each of the three categories. For the present, it 
is sufficient that you recognize the variety of factors likely to be 
present within your own evaluation situation. We have identified each of 
these factors on the basis of our own research efforts and our synthesis 
of the literature on evaluation use. The literature synthesis appears in 
Appendix A. ' ' ' . ... 



When you organize for evaluation use, you will need to examine closely 
your relationship to each of the factors present in ycur evaluation situation^ 
-As an administrator, you. can exert a certain amount of influence over how 
these factors are likely to affect the.process and the outcome of the evalua- 
tion. Just how much influence you can exert may vary with each factor and 
witheach individual situation you encounter during the course of the 

evaluation. - --i 

V. ' " -^^^ , 

V Let's take just one example of a human*^t3>/ ..^^othe one weWe termed ^ 
"evaluator credibility." This factor involves the extent to which, the 



users believe in the evaluator's competence- and In the accuracy of" the 
information he/she reports back to them. What can an administrator do 
«.to^inc4:easo---tbe-^va44jat'CH^*-^ — ______ : 



One step that can be taken is to circulate to the potential users a 
summary of the evaluator's professional accomplishments, including perhaps 
samples of eviiluation reports he/she has done. Another possible step is 
to hold afr>initial meeting with the evaluator and the potential users so 
that the/ can get to know one another better before the evaluatioa actually 
begins. A third possible action is to have past users of the evaluator's 
services communicate with the new_client--user~s,.^^ are several 

things an administrator can do to exert a positiveN'nfluence on users* 
perceptions of the evaluator's credibility. ^ X 

It is important to recognize, however, that some factors are simply not 
amenable to attempts at positive influence. For example, an administrator ' 
can do little or nothing to change any external requirements imposed upon 
the evaluation by a funding agency. 

Nonetheless, you should not be too ready to conclude that there is 
nothing you can do to influence a factor and thus increase the likelihood 
that the evaluation will be;used. Rather, you must try to think creatively 
about each of the factors*; such thinking may suggest a number of strategies 
that will prove valuable. 

In the next chapter, we will define and discuss each of the factors 



whichVthe^^dministrator needs to consi we will 

"^^^""poTntlwt^^^^^ interrelationships among factors, the kinds of infl 



uence an 



administrator can exert over a given factor, and the ways in which an 

evaluation can be organized to take advantage of a given factor's 

potential positive effect on eVeflirattOTnrs^T- — — — — ^ — ^~ 
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CHAPTER III 
FACTORS AFFECTING EVALUATION USE 



In Chapter II we described evaluation use and briefly mentioned some 
of the factors affecting use. While Chapter II was primarily cincerned 



with providlncLbackgrQiind to help you decide who will use the evaluation 
and how they might use It, this chapter will help you with the/ next 
decisions you must make: Which factors are likely to affect uU in your 

project evaluation? and What can you do to make that effect pisitive? 

■ ■■ I. - 

There are three categories of factors you should considei*. 

First, human factors : This category Includes user ind evaluator 
characteVistics which might have an effect on use. / 
Second, context factors ; This category reflects the setting of 
the project being evaluated/ including organUatlonal and 
programmatic arrangements, and social and politica/l climate. 



Third, evaluation factors : Thi^ category refers to the conduct 
of the actual eyaluation, including its procedures, the informa»tf' 



tion it cpTlect^ and how that i>ifonnation is rep^^ 
Nature of Factor Inf 1 uenc^ V / 

The factors discussed here were identified from our /synthesis of the 
\evaluation use literature (including our own research on /eva^^^ 
/use) . The synthesis (Appendix A) col 1 ates various reseatch findings and 
I theories about factors affecting use and offers 'soma 
why a given factor may work for or against use. 



In this chapter we will describe the factors more ^Fully, talk about 
the effect they may have on use, and offer broad organizing tactics, that 
you. can use to create an atmosphere which will encourage use. Chapter IV 
provides information on more specific tactics you may adopt or adapt to 
promote actual use. 

We'd like you to keep one important consideration In mind as you 
read the factor descriptions. Many of the factors In a given category 
are highly related and interact with each other. In the human category, 
for instance, the role an evaluator-selectS-.bears_QDJh„Qwji^^^ 
him. In the context category, an evaluation's contractual requirements 
may determine the extent to which a project can raise its own evaluation 
^ questions and suggest ways to answer them. In the evaluation category, 
the frequency with which potential users get good information can affect 
their tendency-to find such information relevant and appropriate. 

In addition, a factor in one category may be ass'ociated with and 
affect a factor in another category. For example, , a factor in the human 
category, such as the evaluator's politicaV understanding of the project 
(human category) is associated with the kinds of information routinely 
used In a project (context category). Moreover, the evaluator^s pell- 
tical understanding and his appreciation of infofmatioh routines will 
have a bearing on his approach to factors in the evaluation categpry. 

Depending on how these human, context, and evaTSation factors work 
together, their combined effect on use may be positive or negative (or 
neutral, of course). For example, a politically sehsitlve'e^ 
Is aware of users' routines will be more likely to employ procedures 



that are appropriate to the job at hand and thus to provide Information 
with poteijtial for use. 
Application to Your Setting 

Some of these factors are positively related to use In most project 
settings* Others depend on the nature of the particular evaluation set- 
ting. One of your concerns In organizing an evaluation is to decide the 
current status of your project with respect to the various /actors and to 
consider ways to nibdify that status in order to'^ assure a greater llke- 
lihood of evaluation tise.. , ' 

The descHptio^nr of the~1(?d1vidual factors wITl- gi ve-you-a basis-for 
; comparing your setting with a given factor description. What the de- 

scription says about the factor's relevance ^'to use\ and how your setting 
' compares with the description, will help you decide where and how ,your 
influence may be needed. 

For example, you may decide that there are some evaluator and user 
Characteristics you must modify in order to promote use. Now, because, of 
your administrative role in the project, you may readily see ways of in- 
fluencing most users (especially those in your owrt project) but have some 
reticence about the extent of Influence ypu might have on the evaluator. 
But remember, you are not trying to Influence the findings of the evalua- 
tion; .Rather, you are trying^'to act ^n w^iys that will make the eval^ 
tion potentially more usable. It' is certainTy appropriate to try to ' 
Influence all of the actors In order to maximize potential use It may 
be that the evaluator is alrea^^^w fthat 
help promote use. It may bi that you and your c6l leagues are willing to 
work to promote evaluation uW. On th6 other hand,'' som^ individuals^ 



be less committed to use. If so, you may have to work hard with these 
Individuals In organizing for evaluation use. 

A popular song from another. era_sumsuip-a-:^^centra-l^oi^^^ 
Ing for evaluation use — "accentuate the posltlvt^; eliminate the nega- 
tive." The message pays off, so you might keep It in mind as you begin, 
first, with/ the human factors below. 



HUMAN FACTORS 



In this category, there are two classes of factors which can poten- 
tially affect evaluation use: 
Evaluator Characteristics 
User Characteristics. 

INSERT 1 HERE . 

The eval uator characteristics affecting use consist of: 
/. Commitment to use of evaluation results :^ 

Willingness to Involve user/ in the evaluation / 

Choice of role V 

Rapport with users 

Political sensitivity 

Credibility 

Background and identity 
Let's consider each of these characteristics in turn; 



- INSERT. 2 HERE 



/ The ev^;|^tor*.s commitment to use can have a ^strong bearing on /the 
extent to which\results are actually used. The results referred to here 
can be the results of data collection and analysis which are reported as 
findings, conclusions, and recommendations. The fact that they are? re- 
ported to someone does not, however^ guarantee that they will be used. 
But they are more likely to be used if the evaluator isr pers<)nany cpm- 
m itted to seeing his work used by someone. This commitrient is seen in 



the amount of effort — personal, technical , artUtic, persua^sTv^^^ 
9^ ARTomot^^ the use of this work> Of all the evaluator character- 
istics, commttment may be the single most important determinant of use. 

What evaluators feel toward and do about assuring the use of their 
Evaluation says a lot about how they view their craft. Some evaluators, 
expert though they may be, believe that if they do a technically sound 
job, their work Will be used. Other less expert evaluators have ^^^t^ same 
naive belief. Still others do not care if atiyone uses their work. But 
most evaluators want their work to be of practical bene;fit.v 

A potentfal userVjs belief that evaluation resu^ sound helps ;to 

create an atmosphere conducive to use./ And we would hppe that any evalli- 
ator copii tted to use prbvi des sound work . ^ But s^ ng niore 1 s^ re- 
quired. : The evil uator; must of fer a liangible; visible . 
dismonstration that he bel f eves his wor^k shbul d be put to u 

The evaluator needs t^^ show that he believes his: work is useful and 
that he' i9 willing to put it ta use. After all. If people are expected 
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to use evaluation, but at the same time the evaluator does no more than 
pass out "forms" to be completed and send back "data," response is apt to 
~-ixe-->ess~t^afh-eftt-hu5+as-t4<^. — On-4^e.^>^er~~han<!l, if the evaluator solixltii — 
potential users for their questions, gives them timely answers, makes 
himself available to staff members for discussions about progress, offers 
specific suggestions abput how findings can be used, and so forth, then 
he is demonstrating his cbTrmiitment to use and can expect to generate 
greater enthusiasm from potential users. | ^ 

- The research evidence shows that deliberate evaluator efforts to 
Stimulate use tend to enhance evaluation use. These efforts should be 
guided by the Interaction of factors In^^y^^^^ interaction 
that should guide all efforts, yours and the evaluator's, to stimulate 

" As evaluation organizei^ one of youir first jobs is to find out how 
much the evaluator is conimitted to use and what he is willing to do to 
help promote use. 

INSEfrr 3 HERE ^^^^^^^ I 

One way to promote is to Involve potential users in the evalua- 
tion. This Ffjeans that^ potential users can help to guide,, formally or 
4nfonnal1y, the plf.:Vn1ng and condu^^^ of the evaluation as we^l as its ^ 

--uses. ^ ■ rV, ^ , v-'-'., 

Evaluatov:s differ in the extent to which they are willing to involve 
usersf tn the evaluation. On the one hand, some evaluat are less 

• cooinUte(!i to use are also, as we'd expect, less concerned about user 




involvement. If you have^this kind'of evaluator, you should try to get 
him to involve the users it the same^time that you try to increase his 

' ^ •■ -•■■7/ :* \ ' . 

more responsibility. ^' ' ^ 

On the other hand, sdme eva'luators strongly committed to use may not 
see user involvement as desirable, even though we'd expect the ppposite. 

. '^^-^^ ' ; ■ 11 • ■ ' . • ^. .• 

Why should this be so? For one thing, they may believe Jtheir sheer 
rhetorical or statistical per/suasiyeness will issure that their work is 
used. 

-■»> 

11 . , - 

For another, they may fiel ieve it is their exclusive professional 
responsibility to see that^ their work is used. What do you do with this 



kind of eyaluator? Firsy/of all ,^you have, to be careful . You must" make 
*sure the evaluator stay s/conmiitted to use but sharis the responsibility 
by.involving his usersi This can be a delicate situation. But unless it 
is resolved, the evaliiktor's work remains essentially a monologue. And 

such a monologue often goes unheeded. 

•/ ■ ■. .V..- - , ., ■ ■• .- ■ . 

To extend this^ analogy , a user may beseen as an audience or part of 

'/ ■ ' ■ . f . V 

an audience, and ari ©valuator as a performer. Some performers are con- 
tent if the audience buys a ticket, is rgasonablypplite, and clears the 
hall quickly. At best, this kind of .performance evokes only a passive 
audience response. 

Other pjiifi'ormers feel .th^ have not^ d 
u.iless they^ Involve their audience and elicit ^^^^ 

perfonnance. This kind of; perfbrmer probably^ u his abi- 

, 11 ^ to generate an active response depends on the numbep^ of peopl e in 
the audience, their similarities and .differences, the larger communis 



they represent, the political andJsocial attitudes they bring to the 
hall, and so forth. This kind of performer emphasizes audi enc^ particl- 
pfatlon — via empathy, identification^ prompting — in the performance. 
The principal element is a sh&ring:between {partners In the performance, 
the jirimary intention Is to /Vead" the audience a^ them some- 

thing that they can use as they reflect on the performance, relive and 
share it with others, perhaps apply the experience in some aspect of 
their lives* ' ^ 

We subscribe to the Idea of an interactlvfi evaluator-audience rela-/ 
tionship, a relationship that demands involvement and dialogue growing/ 

\ ' ' . - ■ ■ ' ■■- ■ ■ '^.^-^^ - : "^^H • ■ . / ^'^ 

from a shared experience, a relationship that stresses partnership ^ 
promote evaluation relevance and use. ; 

STome eval uators , though certainly not al 1 , share^ our v1 ew./ They 
stress partnership. Involvement, and dialogue In the Interest/of evalua- 
tion use. If you have or can find such an evaluator/ youy^sk of organ- 
izing for evaluation use will be that much easier. y 

• Some evaluators, perhaps unwlt'tif!gly, takerthQ monologue approach to 
their work. They often seem to be talking to "themselves or, at best, to 



^another evaluatoi: or researcher. ^ 

other evaluators deliberately avoid audlenc^ Involvement. This 
avoldajice^moy stem from the "professional 1sm"yjust mentioned, the ten- 
dency to see the audience as naive amateurs who are Incapable of under- 
standing the subtleties of evaluation. Or^ It maty stem from ^^^^^ convic- 
tion that it Is unfilP:to ask already busy peopled help the evaliiator 
do what he is being paid to do. Othe^^ uators onay believe that eval\u<* 
a tor and audi ence shoul d not work together because such° col 1 aboratl op 



.will threaten the evaluation's "external objectivity." These evaj-uators 
may themselves believe, that if users help in structuring their own 



evaluation, the results are "necessarily suspect. , 

Our purpose here is to point out that an evaluator's beliefs, abbut 
the desirability of user ihvolvement depend on several considerations. 
And the' causes of an evaluator's apprehensions about user involvement 
Should determine your course of action. We have ali^eady mentioned some 
possible causes: lack of conmitment to use; belief in the magical power 
of evaluation to persuade and so to be used; notions about 
"professionalism." ' 

vhat are your options for^vercemtngrfthese various kinds of 

reservations? In the case of the "lincommitted evaluator," you^ can " 
explain What kinds of questions you have, what your information needs ° 
are, and how and by whom the evaluation will be used. (You should do 
Ws regardless of the kind of evaluator you have.) You can point, out- 
that potential users will probably take their tue from the evaluator. 
That: is, if he were to ask a potential user how the evaluation can help, 
he might hear something like: ^alk to me about what ny questions are 
and what Information I need to answer them.. Get that to me when I nee'cl 
/it, and I'll try to pu^ It- to use." r 

In the case of the "magi clan-evaluator," you can point put that for 
some people, statistlcs-jn^ jump out from the page, full of meaning and 
rea<iy to be put to use . But not for most peopl e . Most peopl e w1 1 1 1 ook 
f otj expl anati oris, stjggesti ens and recOrmiendatl 6ns about wh^re and. how to. 
use» the f Indlpgj^; ? ^ • ■ „• - 

willingness, to' jsharelthe load, to assure that the Information will be 



used. You can also point; out that- the users need to be similarly 

■4j%V.alV^iL - ^ . : : _™_.__.,.„ 

. These suggestions -grow from one of the important findings to emerge 
from research on the topic: namely » that evaluation use^s enhanced if 
the evaluator actively invq^lves potential users th his w^ft» And this 
Involvement requires, at the very least, a dialogue about the evalulati on 
"process and about how this process, and the information it generates, 

can be made useful . ■ ' 

J- - 

■ ^ ■ ' ^ ' ' - w ' ' ■■ ^ 

" — INSERT 4 HERE — 

if . . . s ■ 

The role that the evaluator chooses, out of several possible roles, 
has an Important bearing on the ^tent to which his work Is used. His 
choice of a particular role will tell potential users a lot about his 
attitude toward his profession,* toward the project; toward the people In 
it, and toward the people who exercise authority over It.^ 

^ • . o - - -i • 

The evaluator' s choice of role 1s closely related to the two factors 
just discussed: conroltment to us^ and willingness to Involve^users In 
the evaluation. Indeed^ both these characteristics can be viewed as 
elements or aspects of an evaluator* s self-defined role. ^ 

There are many possible evaluator roles. One is that of the neu- 
tral, detached, and possibly aloof arbiter. -WMle the evalu may 
choose this role In trie Interest of providing an Impartial evaluation , 
users may get. a vet7 different Impression. In Its most extreme forsn, 
this ,role (evokes ,the Image of «»e "hanging judge,' who does not consider 
■ extenuating clrcumstanc^es In hts rulings. 



/ 



Another conimon role is that of dedicated research scientist* The 
-evaluator choosing this role is Interested in running an "experiment,", 
imposing^a rigorous design, and t:€?stipg hypotheses rather than answering 
project questions arid meeting decision needs. 

Yet another role is that of projdct advocate* The, evaluator who 
chooses this role acts almost as a member of the project staff. He gives 
the impression that he will report nothing negative," for fear of offend- 
ing, or Jjurting the project, or of not being rehired. ^ Perhaps he will 
■ . . \ • . \ - . . . , > ■•■ 

not even make concrete suggestions about- ways to improve the project. 

Perhaps the ideal evaluator role- is that of the colleague who at the 
same ^i me is willing to be cbnstructiyely critical, whowill work with - 
the project staff to raise questions about the project, collect i nforma- 
tion, and analyze it in order to- make honest and helpful recommendations 
about project practices. This role' stresses evaluator-user colTabora-- 
tion. . ^ ^. „ * , 

other evaluator roles may exlH. But almost all th^ stujlies (dealing 
with this, factor agree that- the .last-mpntioned role is most conducive to 
evaluation "se, • . , : . ' ; 

. That evaluators can adopt different rc es and' have <i1fferent°atti- 
tudes toward those role§ is a fact that has Implications for ;iyou as you 
organize the evaluationV Whien you talk with the evaluator about his \ 
orientation to use ahd user i nvolyement,. you should also^iscuss the role 
he will play and the desirability bf hi? adopting a role that emphasized 
evaluator-user interaction arid coll abofatio;). 
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— INSERT 5 HERE — ~ > ^ ^ - 

The* evaluator's ability to generate rapport with u^ers depends on 
the degree of harmony, trust, arid agreement that exists between them. 
Good rapport between evaluator and users is marked by a sympathetic rela- 

tion ship* . / ' ■ ' - ^ - - ' " . ■ " i-. 



Rapport, and its elements of harmony and trust, can grow out of the 
evaluator's professional or persontil approach to his craft, or both. The 
professional aspects bearing oh rapport include the evaluator' s showing 
concern for the needs of "the project, making an^^ pro- ' 

cedures appropriate to the project setting, and ma|cing sure that user ""^ 
viewpoint are invited and Included in the evaluate ^ 

The personal and interpersonal aspects bearing on rapport can take 
several forms. For example, rapport is advanced by ^ the eya^^ 
ability to demonstrate expertise without appearing aloof of haughty and 
by his atcessibinty.. It is not enough that the evaluator be 
to answer questions, interpret, and ^recommend. He should be available in 
^^sfiSf^^ay that users ^^^^^ take advantage of his cki lis. Users should^ 
hbV feel that when they want to talk to the evaluator they must first 
remove their shoes because they are approaching the "presence." . 

As. with the other factors, evaluators vary in their ability, and 
thel r des1 re, to establ 1 sh rapports But s1 nee research 1 ndi cates that 
eval uator-user rapport al so contributes to use, in your organizing work 
you should pay careful attention to this factor. Consider if there are 
ways In which you can "bring" the evaluator into the user group and help 

■• ,.-,>;■:, _ , .-36- ■■ . • 
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to establish rapport. Is there advice that you can provide to the 
evaluator? ' - 

' INSERT 6 HERE^^r ; " 

The potential for use is greatly increased if the evaluator has good 
--^political sensitivity. At least two major e\eFnents are involved in the 
evaluator's political sensitivity. First, hej must understand the various 
aspects of the political context, both individual and organizational . A 
project involves individuals ~ people who differ in beVi^e^^^^^ 
openness, power. Further, projects have decision-making roles and fre- 
nquently exist within a larger organization and are subject to pressures 
wfthin thatv sphere. Thus, in order to be effective the evaluator must 
understand the role of various Important political ingredients such as: 
formal and informal sources of power, trusted opinion-makers, consensus 
'versus dominance^ the Importance of tradition versus receptlveness to the 
new, and breadth and representativeness of the constituency. 

Second^ tne evaluator must understand that information Is only one 
of the Ingredients In the recipe guiding a project. Decisions are made 
on other bases ; atti tudes are formed 1 n other ways . Further, eval uati on 
provides only one particular kind of Infonnati on, a k1 nd ttiat maw not be 
f aratl 1 to all potenti al users . 7 Many sources of y f nf ormatl on are aval 1 - 
able to potential users beyond that provided by the eyal uati on. v 

Eval ua tors may dif f er 1 n thel r abl 1 ity toV grasp th^i polltl cal nature 
of the project, for things. political are difficult to gauge almost by 
definition^ Evjjluators, perhaps as a.funct1on of. their. view of the 



evaluation role, can also differ in their willingness to understand and 
apply political considerations in their work. Nonetheless, evidence sug- 
gests that the evaluator's ability to gain and apply such understanding - 
contributes to use. Part of your organizing work, then, may involve 
convincing th^ evyuator of the importance of political sensitivity and 
helping him to gann an understanding of the political aspects of the 
project. / 

— INSERT 7 HERE — 

If an evaluator is to be credible, potential users must regard him 
as competent, honest, ^nd reliable. Credibility Is a reflection of the 
amount of trust and confidence the evaluator inspires. This trust is la 
part determined by the various technical and professional capabilities^ 
the evaluators brings to the task. In p^rt, also, evaluator credibility 
Is associated with rapport, with personal approach tc the work, to users 
and other factors • 

For example, a highly qualified evaluator may acquire credibility 
because of his reputation for or demonstration of technical competence — 
such as in statistics or research methodology. The technical competence 
associated with content-area expertise is also a source of credibility. 
An eval uator's credibil ity may al so be enhanced because of his abi 11 ty to 
provide timely/ relevant Information or his perceived reputation for 
honesty. Or, evaluators 'credibility might be enhanced by personal and 
other characteristics such as presence and general demeanor or profes- 
slonal reputation. 
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ators, then, can- differ on the^credibiim they bring to an 
asslghfflant. But, Initial jc^edlblllty status Is modifiable, 
research lias shown that^e of „ the- most features of 
crediblHty is the ex^nt to which it m^y be acquired or 



enhanced during the conduct or the evaluation. I 



Thus/ while an evalu^tor's credibility is aij important consideration 

can be done during the 
And, you can contribute 
to/helping the ev^fuator to take actions that willl enhance credibility. 

— INSERT 8 HERE —N 



for th^ulti mate success^ of the evaluation, much 
course of the evalua^^on to enhance credibility. 



Evaluator background and identity refer to characteristics such; as 



age,/ gender, title or position, and so forth, 
that such factors, in and of themselves, usual 



on use. H^. believe that it 1s not background andc¥dent1ty as such, but 
the way ukers react to these characteristics, that has a s'troh^^^^ 
for affecting use. 



Seveiral considerations are involved here. 



evdl uator 1n different ways depending on how he or she embodies their 
beliefs and preferences about age^ gender, race, speech patterns, <phys1 



The literature points out 
y have a limited influence 



Users may react:ito an 



1^ 



^1 



cal attributes i and so forth. Any of these mayl, depending on user reac- 
tion, have some effect on use potential. It Is very likely that — if 
compounded by poor role selection, rapport* political sensltf^ltyr-aTrd- 
credibility — they will .converge to have a iparklpd effect on use. 

The two background characteristics most freljuently studied In re- 
search on evaluation use are gender and title or l^bsltlon. 



Gender, per se, 1s unlikely to have a profound bearing on use; 
except in certain project circumstances. For instance, if users are 
judging unfamniar material , or examining 1 nformati on from a subject area 
about which they have little knowledge, they might mask their own lack of 
understanding and disagreement with findings in the form of stereotypical 
responses. We don't think there is great frequency or likeilihood of such 
occurrences but you ought to consider such possibilities in your own 
situations'. You may, for example, need to prepare users for the informa- 
tion they will be receiving in order to provide a better context for 
understanding^so^tl^^ " 
based on aval uator gender. (While there has been no specific evaluation 
use research related to other kinds of stereotyping, the same considera- 
tions may well apply.) r 

The person evaluating a project can have various titles or hold ^ 
various designated positions: eval uator, researcher, monitor, consult- 
ant» and so forth. Depending on! a user's prior experiences with an eval- 

. ; . i ■ ' 

uator, external consul tiant, etc., the title or position of a given evalu- 
ator may bear on how his work is received. Should users be predisposed 
against a certain title, you may need to present the evaluator in some 
other way. 

Again, title by itsfelf probably has little effect on use, except In 
such circumstances as those described above for gender. Your primary 
organizing concern in connection with title and gender— as well as 
other personal character! sti cs of eval uators 1 s to mi nimi ze the extent 
to which users' expectations and prejudices will cause them to react 
negatively to the evaluator and because of that, to the evaluation. 



On the basis of ap of the above evaluator factor descriptions, 
there are a few questtiiiins you might mind as you consider how to 
organize your evaluation. 



' things TO THINK ABOUT 

What are the characteristics of t^^^^ have . 

worketf with In the past, or may be working with now, com- 
pared with those described above? 

How do you and your colleagues, as potential users, stack up 
In terms of becoming active evaluation partners? 

How difficult do you think It may be for an evaluator work- 
ing with you to assume a project-sensitive role? 

Would an evaluator of your project have difficulty under- 
standing the political situation? 

What evaluator personal characteristics would enhance or 
detract from credibility of your project? 



As you consider these questions, and other relevant questions you 
yourself may pose, you may be helped by using a scale like the follow- 
ing. On each factor described above, try to pinpoint where an evaluator 
that you have worked with, or that you are working with, would fit on the 
scale. Where you are dissatisfied with a factor fit, consider what you 
might do to Improve -It. . 



^ will have no 

will hinder use , effect on use will promote use 



~ INSERT 9 HERE — 



The user characteristics affecting evatuati on use consist of: 
Identity 

Interest in the evaluation 
Commitment to use of evaluation results 
Professional sty! e(s) 
Information-processing preference(s) 



—INSERT 10 HERE — 



Identification of users presents a logical starting point for begin- 
ning the examination of user characteristics. This factor includes a 
consideration of who are the various users, what are their organizational 
positions* and what are their professional experience 1 evel s* Let' s 
examine each of these elements in turn • 

An evaluatiort may often n6ed to be directed towarjls multiple users, 
or groups of user s> At the outsets you^houl d provi de the eval uatbr wi th 
a clear picture of who the ' potenti al users are and di sicuss how each might 
use the evaluation. Once the eval uator has a clear picj^ure of the users, 
he is in/a better position to decide on the best evaluation processes and 
techniques, and generate and report Information which ^^^^^^^ 
pptential for use. 

The various users are likely to have differing organizational 
^CfSlMohs^nd 1^ 



tionaV positions while others will have important support functions in 
the project. While many of the potential users will be found inside vbur 

^ , .. \ _ . ^ - ' ■ . -■ ..y-/- ----- - ^ 



project organization, others might be located in other agencies or even 
the community. It is up to you to give the evaluator relevant informa- 
tion on this point to help focus his attention. (In this ha^^book, we 
will focus primarily on the users within the project organizationjN^ 
you will be able to easily extend our comments to other groups of users.) 
...V You should also consider the professional experience levels of the ^ 
potential users. For example, there will be differences in length of 
service with the organization, in the nature and length of the training 
that each of the users might have received, and in the amount of time 
they have been engaged in their professions or occupations ^ These dif- 
ferences have implications for your evaluation organizing. A person who 
has mastered the responsibilities of hts present position moy have a 
greater potential, for using evaluation. Conversely, someone who is 
devoting all his energy to mastering a new Job may find it difficult to 
devote any time to evaluation use, especially If iuch use is not related 
to his normaV routine or not_a_ part of his repertoire of prior 
experience. 



There 1 s som&rflkel ihood that those wi th the greatest invol vement 1 n 
the decisionmaking process/ and those with the greatest e^^ In , 

Sjelr positions, tend to be more critical of the evaluator and the evalu- 
atlon than those with less decisionmaking ^authority, and ^experience. This 
attitude may have several explanations. A person who has been making 
decisions for a good part of his professlional life, without ybenef It of - 
formal evaluation Information, mv feel no need for such Information. 



As evaluation organizer, one of your jobs is to help an evaluator to 
jjndejrstand the ch arac teristics of the^e potential users • Knowing who the 
users are, how they resemble or differ from one another, and where they, 
'stand oh the various factors discussed in this section will help the 
evaluator in planning, the evaluation and developing appropriate proce- 
dures and strategies. 

/ — INSERT 11 HERE — ■ 

Like user identity, user Interest In the evaluation has several com- 
ponents: views about the project being evaluated; expectations for- the v 
evaluation; predisposition towards evaluation; perceived need for the 
evaluation; and perceived risks of the evaluation. Each of these 
Influences the extent of Interest potential users will have In the' 
evaluation. 

Potential evaluation users can have different views about the pro- 
ject being evaluated , and these views — consi sti ng of atti tudes toward,, 
beliefs In, and expectations about the project 

potential of the evaluation. For example, some users will be neutral , 
holding no particular brief for or against the project. Other potential 
"^users will have f1m views, pro or con, about the project's value even 
before the eval uati on takes pi ace. H1 th1h th1 s 1 atter group there may be 
some Individuals whose copn1 "bent to -a, particular view 1s so s^^^^^ 
Uhey cannot modify it, regardless of what the^ evaluation shows. They 
will maintain their original position even if changes are warranted and 
beneficial to the project; 4s0thers may be more flexible, more willing to 



iodify their ini^aV views on a project in Tight of the evaluation 
f i ndi ngs . - 

In conzidering the view of the various users, you will need to make 
a number cf decisions: \an you expect users with initially firm views to 
modify th iews if warAfited by ^valuation findings? To what extent 
will their original views/ of \the project affect.,their approach to evalu- 
ation use? 

\fi ^ ' 

Users differ in their expeditions for the evaluation — what they 

• > ' \ ' 

consider^that-the-evaliiatlon wf 11 produce. Included in the expectations 
are the specific questions and concerns of the users and the levels of 
urgency attached to these. 

Users outside the project may be expected to have different ques- 
tions a^nd^ concerns about the project from those who s^re part of ,the pro- 
^ject staff. And, even potential users who work in the same project, or 
h^ve s1m11ar'r<^pons1b111ty within the project may have different ques- 
tliOns and concerns about the project. Some pbtentiaV users may be 
primarily concerned about the efficiency of the project \s operations. 
Otners may focus primarily on' the kinds of effects the project Is 
haying. Still others may have questions about both areas. In addition. 



the questions and concerns that an uninformed evaloator regards" as rele- 
vant may differ from those viewed as relevant by users familiar with the 
project. Early discussion .of the range of possible questions and 
concerns, along with a frank airing of differences on ^^^^t^^ mi^y 
prevent 1 ater misunderstandl ngs and create an atmosphere conducive to 
use. . . • • * ; 



Users may attach different kinds and degrees of urgency to their 
questions and concerns. For some users, urgency involves the press |of 
time; they want Information quickly. For other users, urgency may refer 
to the gravity or importance of the situation; they believe that unless 
a specific questTorTTs answered and answered fully they cannot fulfill 
their responsibilities. 

Expectations may be viewed^ by users in other ways. For example, 
some users may want to know how well they are doing the^ir jobs. Others 
may want the evaluation to yield concrete recormnendations they can apply 
to their present practices. Still others may expect that the evaluation 
will merely give the seal of approvaV to present project activities. The 
expectati ons of others may be vague; they only want the eval uation to 
encroach as little as possible on their time. In general , if eva'^uati ons 
do not match potential users' expectati 6ns, they are not likely to make 
use of them. - / ^ 

These differences in expectations are to be expected. Like other 
differences among users, they have ramifications for your organizing 
efforts. For example, if the evaluation entails only a few questions,^ 
the fact that they are all urgent need not create a problem. The 
eval uator, with your guidance, may be able to distribute his talent and 
energ1es\ among them. . But many urgent questions, especially if they are 
spread acro^several different kinds of potential users maor create a 
dilenma for you and the eval uator. And if it creates a dilemma for you 
and the evaluatar, it is likely to create a dilenma for other users as 
wfeil, who eXpect Information matching the urgency they attach to the 

: •■ • ■ ■ ■ ■ ■ • . . . ... .. . : - 

question. ■ ■■ 
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If it seems impossible to respond equally to all urgent questions, 
you may hs- : to consider rediiaing their number,' perhaps by collapsing and 
c- jining lonie by eliminawTg^some, at least for the present time, 
i.. 3ther u' estions are|reduced in number or not, they can at least 

be priori ^i^eu. • '^x 

In short, the evaluator will need to have a clear sense of thfe range 
of expectations. You may have to work actively with both the evaluator 
and the other users, to clarify these expectations, emphasizing some while 
downplaying others. In short, you can. set the 'tone for what the various 
users will expect from the evaluator— modifying some views in the 
process — and help the evaluator to better understand what exists. 

Just as users can differ in their expectations, so they can differ 
In the predisposition — the inclination or tendency for or against — 
that they bring to the evaluation. Negative; predispositions will 
undoubtedly inhibit use- potential ; thus, it is important to be aware of 
and attend to these predispositions. Likewise, the evaluator will want . 
to be aware of and capitalize on positive predispositions. 

For example, you will have to try. to modify the position of those 
potential users who are disposed against evaluation. You may be helped 
i n your efforts wi th ;^se negati vely di sposed towards eval uati on by 
f i ndi ng out the reasons for this atti tude . Some users may thi nk that*anr 
eval uatlori 1 ooks only for faults . Others may have been hurt in some way 
by a previous eNfdl uati on. -'Qthers may think they wi IV ^b^ devote 
large chunks of their valuable time to the evaluation. 



In the case of users who are favorably inclined toward evaluation In 
general, you must make sure that the parti cul sir evaluation is carried out 
in such a w$y as not to damage that inclination. 

The use potential 0/ an evaluation will be increased if users see a 
perceived need 'for> the evaluation to be conducted* thus, users may feel 
that the evaluation will help them to find out how well the project Is 
doing and, how they can improveriheir performance in and contributions to 
the pro j ec t • iuch users are 1 i kel y_-tO-makeluse_of_the-eyalu atlon^*^^ 

On the other hand, some users may see little or no need for an eval- 
uation. They may think It Is too soon In the life of the given project 
for evaVwicion, that the project should be allowed to develop more before 
It Is conducted. Or they may think It Is always too soon, that the money 
and time could be put to better use In other activities. 

Again, differences In perceived need for the evaluation can lead to 
^different use potentials. The kind of suggestions just offered with 
respect to predisposition may also be helpful in considering ways of 
improving prospects for use related to the perceived needs of users.^ 

Just as users can differ on their views of the perceived need for 
the evaluation, so they can differ on the perceived risks they associate 
with the evaluation. Indeed, the two are related. For example,\some" 
users may fear that negativdi^ findings can threaten their jobs or chances 
for~promot1on. Others may fear that evaluation recqmmehdatlons will lead 
to broad and sweeping changes that 'wlYTr-^n^^ -Still 
others may fear comparl son with their p.eers. 

Perceived risk can Influence the extent to which people are likely 
to use the evaluation. Given Its possible effect on use, .your organizing 
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efforts should addresls.the question of risk. One of the first things you 
might do is to consider whether you yourself associate any risks with, the 
evaluation. If you do feel a sense of risk and if you somehow convey 
that sense to other users ^ there is a good .chance, for reasons mentlomSd 
earlier, that they will also feel 'threatened. Conversely, if you project 
the attitude any possible risk Is offset by potential benefits, they will 
take this cue from you. ^ c vv 

The perceived risk may be overstated. It me^y be unlikely that the ^ 
evaluation, in and of itself, will result in the kinds of profound 
change? that people;fear. - Pferhaps all you have to do is take appropriate 
steps to make sure that possible fears are not realized. For example, 
you should not arl-low the evaluation to provide only a one-time, end-of- 
project report, even if you expect .positive results. One important 
contribution of evaluation is to pinpoint trouble spots as they emerge,- 
before they. become so severe they cannot be tdealt with. If the users are 
aware of this evaluation^ focus, their anxieties can be minimized. 

On the other hand, an evaluation may entail'some; real risks: for 
Instance, the possibility that the project will not be refunded. Where 
such risks exist, you should-apknowl edge them and ^discuss the possible 
range of outcomes. And you, should again accentuate tlie posVtive," 
examining the options that face^youi / ' 

■ ■ ' y* ' " - ' ^ ^ ■ " ' ^ ^ y " """^ 
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We already pointed out^ that :evaluat^^^^^^ is one of ' 

the Isi ngl e ,mo5t important factors af f ecti ng use, and 1 1 1 sn ' t surpri si r\g 



that the same holds true with respect to users. No matter^how much the 
evaluator is coiranitted to use, if that same coiranitment is not shared by 
Intended users, then the potential for use diminishes sharply. This 
nptlon harks back to the metaphor of evaluator and user- participating in 
the same performance. " 

Clearly, we believe that -the stronger the interest in the evalua- 
tion, the greater the likelihood that users wiV! be conmiHted to using 
the results or benefitting from the picocess. Thus, perhaps the best way 
to increase coiranitment is to assure that<;the evaluation is conducted in a 
way that addriesses expected issues, by youf\ minimizing risks and the " 

like. In some instances, however, there firay. be other things you can do 

■ ■ ^ / ..-^ X.' . ' ■ : , ■ ; ■ ■ ■ ' ■ - ■ -.i. . • ■ 

in promoting ^commitment to use as part of your organizing work. 

Potential users are more, apt to become actual users if key decision- 
makers anVl important othe»J'visibly show their support for and active 
commitment to use. Thus; 1t is" particular^ Important to obtain commit- 
ment from those who hold titular power anfli^horlty or who occupy a 
trusted p,osi ti on wi thi n the user group (w1 th or without formal authori ty ) 
or who are known for taking the initiatiye, assuming responsibility, 
"making things happen." The tangii)le commitment of key users has been 
shown to stimulate the potential for use, -though it by no means guaf^an- 
tees that evaluation will actually be used. 

Let's return to the metaphor of evaluator as performer. Assume that 
the evaluator has develbped a rapport and active involvement with most of 

< the users in the audience. They, are beginning to get' something from his 

■ '■■ ' ' , 

work ahd may be thinking about ways to put it to use. But there are 

jT' 

• "ho'tdouts" in the audience, -people who are not caught up in the perfor- 
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mance, who may even belittle the-evaluator and his workJ If these "hold- 
outs" are key. personnel whom others trust and respect, ihey may have a' 
dampening effect on the whole affdience. Others in lesser positions' may 
take their cues from the ke:( personnel and adopt a similar attitude, out 
of fear, out of a belief that the holder of the negative viewpoint is 
smarter than they are, or out of a desire to Maintain harmony. Whiatever 
the reasons, their own enthusiasm and commitment will be blunted. 

Therefore, as >ou consider how to organize the evaluation, you 
should give prominent attention to the question of how to commit key 
personnel to use. You should start by exerting your own influence on 
this potentially valuable group of key people. 

Where there is no commitment to use, and where suggestions such as 
the above do not pay off , then you may need to consider whether it Is 
worth doing the evaluation. -If you have the authority, you sh^ 
perhaps postpone the evaluation until such time as you can elicit some 
commitment to use. / 

— INSERT 13 HERE -- 

Professi onal styl e of users ~ the skll 1 s and attrlbiites associated 
with the way they conduct thel r work -- will be considered next. Several 
aspects of prof esslbnai styl e to be considered are: a<^^ and 
organizational skills; Initiative, and openness^to new 

Potential users can differ In thel r >adm1n1strailWj^^^^ 
tlonal -skills . For example , you may havf ablllty:;^^ y o 

istarted and to f ol 1 ow through and get them done; But some of your f el 1 ow 



administrators, who are potential userSr jnay be perpetual procrastlnators 
or otherwise unable to get tasks finished. Too much of this behavior may 
have a negative effect on the use potential you are trying to build. 

Similarly, other users differ in their ability to organize their 
time to promote use. If evaluation use, as defined here, is a new re- 
sponsibility, for some, you should ask your more experienced colleagues 

how-they-plan~to-orgafi^e-4:he44^!-time-50-^^ 

tlon. their plans can then be shared with others. You will need to 
follow through to help colleague users in following through. 

-While user Initiative is in some ways part of administrative and 
organizational skills, we view it as sufficfently Important and different 
to consider separately. Initiative Includes the abi 11 ty^to take the 
first step, to grasp and develop a hew idea or method. While always a 
desirable attribute, this ability takes on an added value as regards 
use. If the evaluator is actively promoting use in a particular project, 
then user tnitlative is a welcome complement. Whenever evaluatjor 
promotion of use is less than adequate, project level Initiative assumes 
great Importance. 

Users- openness to new Ideas or change can have, a strong effect on 
eval uati on use. How users stand with" respect to any new idea or change - 
may determine thielr stance toward evaluation in particular. For in- V 
s'tance, some users mjay cling to tradition and! may resist innovation and 
change. Others may eagerly seek — or at least accept arid apply — the 
new and different. Others may take some pbsti on betwe^th^ 
extremes, withholding judgment until 'they can make a decision based on 
experience. 



Evaluation may be part of the professional repertoire of some 

■ ' ■ ' ■ ■ ■ ' ' '■*■.'', '\ ■ ■ • ■ 

users. For others it may be something new and different. Within this 
latter group, some may welcome the new experience; others may actively 
resist it; others may adopt a "wait and see" attitude. Agaiai discus- 
sions Involving the evaluator, you and other potential users will be 
important here. As the evaluator and the various users get a sense of 
any differences that exist, t^ arPJ^^nd Js being prepared for evaluation ^ 

-use^^— -V 

/ The organizing tactics previously outlined can pay off here; giving 
potential users an orientation to evaldati on, clarifying the nature of 
evaluator-user relationships, discussing risks^frankly, and enlisting the' 
aid of peer-allies may all help to Increase the likelihood of evaluati bh 
use. - / ■ ' 

— INSERT 14 HERE 

Information processing preference refers» f1rst» to a user's 
preference for particular forms of Information. One user may feel his 
questions are best answered by very fine-grained Information, while 
another may want a broader sweep • HI th 1 n these 1 arge d1 f f erences , some 
users may prefer narrative descriptions; others, statistical presenta- 
tions; and still otheflrs, a combination of the two^^ 
to help the evaluator develop a sense of how parti cular^-<^users' questions 
might ;:est be answered, and you niay need to work with other users^^t^ 
discuss preferences^ modify preferences^ negotiate possible conf lets »andj 
so forth. 



Second, users can differ on how they process Information. Some 
groups may be, used to receiving reports which they can siudy in private. 
Others may be disposed to information which is first presented orally and 
then summarized in written form. Other users may prefer that an 
individual representing their group first receives the information and 
then shares it with colleagues. 

In short, users may differ markedly In their preferences regarding 

-the-form-that-lnformation-takes and the way In which it Is transmitted. 

And research shows that unless the evaluation recognizes and builds on 
these preferences and routines, the use potential may be hampered. 

• You can organize for tfris factorbylsetting up opportunities for the 
©valuator, you, and other users to djscuss styles and preferences and to 
devel op consensus . These d1 scusslons ,1 rici dental ly , can sierve another 
purpose. They may uncover prevtbusly unknown facts about how project 
staff members feel about other things, how they prefer ^^t^^^ 
things.'. Administrative action based on these feelings and preferences 
not only helps prepare for evaluation use but also leads to Improvements 
in overall project operation. 

These are a few questions relating to the us^r factor de^^ 
provl ded above that you mi ght consi der as you organi ze for eval uati on. 



THINGS TO THINK ABOUT 

Win your users' organizational positions and experience levels 
have a good or bad influence on use? 

How do your various users/user groups differ regarding their 
levels of interest in the evaluation? 

How strong a role will you, need to take to promote comrnitment 
-to and actual use of the evaluation? I 

Will your users' interest lev^, skil 1 s, initiattve>-and Ur~-- 



openness tend to help or hinder use? 

How difficult will it be for you to provide the evaluator with 
a sense of users' information preferences and processing 
s^les? 



As you consider these questions, you may want to refer to the scale 
previously provided. Try to determine if the factor, as presently 
manifested among potential users and user groups. 



will have no 

will hinder use effect*on use will promote use 



Now, consider any factor which you think will hinder uses What ideas come 
to mind for influencing that factor to make it work for use? 



Context Factors 



In the context category of factors potentially affecting evaluation 
use» there are three broad classes: , 
Preexisting Evaluation Bounds 
Organizational Characteristics 
Program Characteristics 

As was the case with the human factors /there are some context 
factors you may need to influence to help promote evaluation use. 

— INSERT 15 HERE 

Preexisting evaluation bounds refer to events which may have taken ; 
place, and to Important decisions that may have been made. Which set 
possible constraints on the evaluation. The three factors in this 
category are: 

Written requirements 

Other contractual pbligations 

Fiscal constraints 



INSERT 16 HERE 



Written requirements Include legal codes, federal /state requirel 
ments, institutional operating policy, and specific requirements set at 
the unit being evaluated. If these requirements permit some flexibility 



in tailoring the evaluation, or part of the evaluation, to the project, 
the potential for use increases. 

You, the^evaluator, and other u?ers should discuss the range of; 
requirements. First, you should think about the best ways to meet them. 
Second, you should see whether external requirements conflict with speci- 
fic project needSi If so, it is critical that you try to resolve these 
potential conflicts. Again, you may need to negotiate among the sources 
of the various requirements. If an evaluation tries to be all things to 
all masters, its use potential is severely threatened. Ideally, you, the 
evaluator, and other, users (including external users) should be able to 
resolve any possible conflicts over requirements. And what could some of 
these conflicts look like? 

Quite often, the conflict is less a matter of disagreement about 
what the evaluation should provide and where it should go, and more a 
matter of how it should get there and what it should do on the way. 
Another song comes to mind here. You and other users might agree that 
{he evaluator should meet you in Scotland and give a report on his 
journey. But one important external user wants the evalua'^i^^to take the 
low road, perhaps because he thinks it the most direct route. You want 
the evaluator to take the high road, perhaps because you think he'll have 
a more interesting and enlightening journey. 

Now, with no better directions than~ that, what can the ev«(luator 
do? He may follow one set of directions and satisfy pne^^^^^^^u^^^ 
disappointing the other. He may take one path part of the way, th^ 
switch to the other; and later return to the original . If lyiui're lucky, 
m may send you a postcard from Lodh Ness saying h^'s stopped for a few ; 



days to look for the monster. Thus, by the time the evaluator arrives at 
his destination,, both users have packed up and headed for home. 



- INSERT 17 HERE — 



Other contractual obligations. In addition to the written requi re- 

^ " \ ■ ■ - . \ - ■ 

ments noted above, can be set by project administrative and operational 
users. If handled properly, these locally generated reouirements can 
enhance use, by providing a user-oriented focus and direction for the 
evaluation. | 

Initially, however, project-level obligations may-^be unrealistic or 
overambitious. Further, they may create tension for\ the evaluator, who 
i s al so tryi ng to meet the wri tten rec|ui remehts • Oi scussi on of thi s 
potential problem should aim at producing a manageable set of require- 
ments* You might set project-level obligations in such a w^^^^^^^ 
evaluator answers particular questions for specific users. Evaluator- 
user discussion of these questions can have a posit1v€r effect on use by 
helping users to define their needs and expectations eind the evaluator to 
select the most appropriate role. 

: * . ■ - ■ 1^ : ' 

I ■ • , : - . ■ ' ' . . ■_ \ 

— INSERT 18 HERE — 




Fiscal constraints — I.e., the amount of^iibney j set aside for the 
evaluation— can affect an evaluation 1n>'^ar1ety pf: ways. For 



example, the time the evaluator spends^OD-^- project.' the measures he 



devel ops or sel ects , and hi s 1 i|form^ti on reporti ng procedures are al T 
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affected by fiscal constraints. These constraints, therefore, have. a 
bearing on the kind of role he selects. But while you should pay atten- 
tion to fiscal resources, they need not, in and of themselves, seriously 
impede evaluation use, unless the problem is so severe that it gets in 
Che Way of real attempts to meet project needs. 

The first thing you should do is to- discuss with the evaluator how 
much of his time will be taken up by various questions, needs, expecta- 
tions, etc. A fter converting this tim e ind_ex_.to dol 1 ars, you may f 1 nd_ 
that your own and other users* expectations have to be modified. The 
second thing you can do, if you want to answer more questions, than money 
will permit, is to look for "cost-free" alternatives. You, the evalua- 
tor, and other users may be able to come up with ways for using project 
personnel to perform some evaluation tasks: collecting anck recording 
data, for example. Such alternative strategies may substantially reduce 
the evaluator' s time-dollar conmiltment. 

" INSERT 19 HERE " 

The organizational characteristics potentially affecting evaluation 
use may be examined in two groupings: 

Intraorganl'zatlonal features V 
£xternal features 

The factors In this group reflect patterns^ and relatlonsM within 
and outsi de of the organlzati on that may affect use. Scwne of the 1 ntra- 
organizational features have alrea<^y been touched pn In 1^ 
user characteristics. 



— IHSE^SfZQ HERE — 

■ ' . I ,■■■«' 

Intraorganlzational features refer mainly to establfshed character- 
istics and relationships of the organizational unit to which the program 
being evaluated belongs. Examples of such vfeatures are the role of the 
governing agency, the relationship between the program's unit and the 
governing agency and the unit's degree of autonomy. 

Rut simply, the unit with whicfi the evaluator is working Is likely 

to have a past history which can affect both the conduct of the evalua- 
tion itself and the potential for use. Ther*^ 'may' be little the evaluator 



can do to al ter these, past patterns 



of interrelationship; however, under- 



standing how they are Tikely to impact on the present evaluation is im- 
portant to responding to the needs and attitudes of potential users, to 

plans for conducting the evaluation^ and to the likelihood of evaluaticn 

. . ■ . '\ '' . ' ■ " . ■ ■ ■ .■ , 

use. ■■■ i : ■ 

Let's take a closer look at what some of the intrabrganizational 
features can mean to an evaluation. | Oneexample might be the case where 
a project's governing agency generally supplies support services $uch .as 
consultants, in-house expertise, and the like* Such an agency would also 

likely to select the evaluator for the program. This situation will 
havfer definite implications for the evaluation organizer in terms of the 
evaluator's credibility, how the evaluator's personal characteristics and 

\ ■. . ■, •- ■ , . ; ■. ■ ,■; ■ ■ ■ 

operational style are likely to complement those of the project staff and 
the potential users , and the ul tijnate 1 Ikel ihood of evaluation use. 



A second example might be the degree of autonojuy exercised by the 
project. An evaluator who must conform to expectations and requirements 
set by a higher authority within the organization may have correspond- 
ingly less opportunity to respond to the particular concerns and needs of 
the program itself. Unit autorjpmy is more likely to provide the organi- 
zer^ evaluator and potential users a greater degree of freedom- to decide 
how to conduct the evaluation and to use its results. Your concern as 
evaluation organizer is to determine the extent to which lack of autonomy 
might inhibit the evaluation's usefulness perhaps this is a negotiable^ 
item for a new evaluation. 

Finally^ where institutional risks are real> they should' be dealt 
with openly and the possible outcomes of the evaluation should be 
discussed. 

■ ■ ■ -■ 

. / — INSERT 21 HERE 

The external features refer to the context outside of the actual 
project. Included in this factor are such elements as the community 
climate^ the community influence^ and the role of other agencies. \ 

Some projects operate within the larger conminlty; for Example > a 
specialized medical clinic^ or a sports program like Little League. .The 
community where such a project Is located may be supportive^ neutral or 
hostile to the presence of the project. We refer to this coninunity atti- 
tude toward the project as the community cllniate. Of te^^^^ • 
will transl ate 1 ts attitude through attempts at Influencing this oper^ti on' 
or the funding of the project. For example^ parents supportive of a 



bilingual education program may attend cpmunlty advisory council meet- 
ings, may volunteer their support by' speaking In favor of the project's 
continuation, may work as teacher aides, and may sign letters of support 
to be submitted with the project's application for renewal of funding. 

Evaluation can be used to provide the community with evidence for 
continuing to support the project, Thus,_1n your efforts to promote 
evaluation use, you need to Identify those members of the community who \ 
have a stake In the project being evaluated, and try to involve them In ^/ 
the evaluation ^ much as possible* : . : : 1 -.1 

In addition %) the corranunlty's relationship with the project, there 
may also be an externlff=%^ncy , governmental or otherwise, with a stake 
in the project. We have alrea(^y seen an example where an external vjnd- 
ing agency has set criteria for the evaluation to satisfy. Someoutside 
agencies may continue to try to influence the conduct of the evaluation 
as it progresses by changing or modifying some of the -original require- 
ments. To the extent that these modifications influence the kinds of 
information to be collected during the evaluation, they can affect the \ 
potential for evaluation use. You may need to urge the agency to recon- 
sider Its position if you feel the effect ofsuch changes will reflect 
negatively on the potential for evaluation use. 

. — INSERT 22 HERE — 

Many program characteristics can affect evaluation use: the size 
and scope of the project; Its content. Including topic and emphasis; the 
number of people It serves; the >»ay In which they are selected; and so 
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forth. However, the kind or extent of effects that each might have orf 

use is not clear. 

' ■ ^ I .■■.'** 

The three program charcteri sties on which there is greater clarity 
and which have been found to affect use are: Age /maturity, I njiovative^ 
ness, and Overlap with other projects. We will first define each of 

these characteristics and then discuss how all of them, working together, 

* ' ■ . . ■ * 

may affect use* 

Age and maturity are different entities. The age of a project 
refers to how_ long 1t,has_ beenjn p^^^ Is the project brand-Tiew? 

Has it been in operation long enough for some shaking-down to have 
occurred? Has it been operating long enough for staff to be routinely 
performing their jobs? For some of its goals to be accompli sjiied? 
Project maturity reflects the' extent to which project procedures and ' 
expectations are institutionalized. A project's age may or may not be 
related to its maturity,- depending on how maturity is manifested. / 

Project innovativeness can also take several forms. An innovative 
project may be one that embodies something new, or something creative, 
.or something unique, or something with an element of risk attached. 
Innovation shouTd.be viewed against- normal project routine. If somethinjg. 
is added to the projebt that is a marked departure from the routine, then 
the people operating the prig ect are likely to see it as at least 
somewhat innovative, even though the addition may not meet the strict 
definition of innovation. ; 

Project overlap takes three different forms, each of which has dif^- 
ferent implications for your organizing work. The first is the extent to 
which a project .1s part of a larger systemwide or programmatic .effort. 



The second form "is the relationship between one project and others in the 
system that are ndt prograiwnatically related. The third is the project's 
relationship to similar projects outside the system. 

One dominant strand, with strong Implications for your organizing 
activities, links these three program factors age/maturity, Innova- 
tlveness, and overlap — In terms of their effect on use. That is, if 
the evaluation 1s to have potential for use,Mt must be geared to the 
needs of the particular project. These needs may differ based on the 
proigram cha;^act6ri sties. - ["-. ■ * 

Thus, for example, a new project often needs the benefit of forma- 
tive evaluation to help monitor progress and modify practice. iPln'older^j 
perhaps more mature, project might benefit from formative aval ua-^i.on but 
may be primarily stand in need of siipnative Information showing overall 
levels of success. Any information mismatch, especially in the c a s^ 
young project receiving only summative information, will probably have a 
negative effect on use. ) . I ' 

By way of further example, projects linked progra^^^ with 
other projects may be subject 1:o simultaneous evaluations. If the indi- 
vidual project receives Information Which paints only the overall pic- 
ture, then project-level potential for use diminishes. The individual , 
project will be looking for fine-grained information wjiich It can use for 
its own Internal purposes. FurthenSore, a sense. of coinpetition is" common 
among projects linked in some larger effort. In this case, the credibi- 
lity of the evaluator is critical, since the project wants to be sure ^ 
that it receives a fair. and accurate portrayal. 



As you organize for the three program-related factors, you will need 

to consider your project's characteristics on these dimensions. Yt)ur 

. ■ ■ ' • ■■■ ■ ■ / 

major concern is to make sure that you, the evaluator, and the other 

■ ' ■ -J ' ' ' 

users agree on how a given project should be characterized and what 

/■ )■. ■ 

implications that might have for the evaluation procedures. . 

Drawing on the above context factor descriptions, here are a few 

questions that may help with your organlzlng-for-use activities. / 



THINGS TO THINK ABOUT 

Are the pree^stlng bounds such that there may be conflict 
between project expectations and external requirements? 

Is your own authority such that you can ensure that the evalu- 
ation, or part of the evaluation, will address project needs 
and questions? 

Do you think an evaluator might have difficulty presenting his 
findings SjO as 1;o be compatible with routine organizational 
Information patterns? 

To what extent will you have to consider community needs as 
part of the project evaluation?;^ 

How would you characterize your project and Its evaluation 
needs wltfi respect to age/maturity, Innovatlveness, and 
overlap? 



Once again, as you consider ihe Individual factors suggested In the 

above kinds of questions, does It appear that a given factor 

c . will have no ^ 

win hinder use effect on use will promote use 



How can you influence any negative factors to make them, if not 
supportive of use, at least neutral? 

EVALUATION FACTORS 

In previous sections we have examined factor grouping dealing with 
the personal characteristics of the various individuals engaged in the 
evaluation and its use (human factors) and that cluster of factors 
relating to the organizational, contractual and fisc,a1 bounds influencing 
the evaluation (context factors). We now turn to the "Evaluation 
Factors" — • those factors dealing with the conduct and reporting of the 
evaluation. 

.Four classes of evaluation factors may affect use: 
Information Dialogue ' / ^ 

Substance of EvalMati on Information ^ 
Evaluation Reporting / 

* Evaluation Procedures 



— . ■ — — _ — . — — 
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, ^ —INSERT 26 HERE — 

An information dialogue is the purposeful, guided sharing of ideas 

and opinions, between evaluator and users, about how to conduct the 

evaluation to promote its use. This dialogue is purposeful because its 

intention is to influence an evaluation's conduct to enhance its use 

potential. It is guided because evolving tactics to promote use are 

informed by a careful consideration of^^tKose factors which, in the given 

situation, may affect use. * 

... * . , ^\ ■ 

The interaction whic'f fnajces up the dialogue should take whatever 

forms are appropriate to the situation: formal /in formal conversations; 

meetings; or written communications. 

The success of such information dialogue is determined primarily by 
the amount and quality of the interaction that takes place between 
evaluator and users. ^ ^ 

Once again, the metaphor of evaluation as performance hiais 
implications for your organizing efforts. That is, we believe that the 
most rewarding evaluation is one in which the evaluator's performance is 
guided by the users in his audience. The evaluator has to ascertain user 
needs and conduct himself and his work accordingly. But it is a two-way 
street. Users have to voice their needs. Thi^y must be actively Involved 
in the performance ~ interacting — so thatVt^^^^ adopt 
procedures and provide information reflectihg specified us 

Both the amount and the quality of Vnteraction are impo^^^ The 
— amouf^t-of-eva^luati^n^user ftiteraction can\^ 
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and the earlier that Interaction begins, the greater Its Influence. 



Interaction Includes a broad range of encounters between evaluator 
users: making formal and Informal reports, making and taking phone 
calls, requesting clarification, explaining Information, sharing a draft 
before It Is finalized, discussing evaluation progress over lunch. These 
encounters can be initiated by the users as well as by the evaluator. 

The quality of evaluator-iser interaction can also affect use; 
numerous contacts, of themselves, are not enough. Each of the inter- 
actions should have certain qualities. First, the atmosphere In which 
they are conducted must be collegial and reciprocal. For example, a . 
discussion of progress In which the evaluator does all the talking is 
probably not desirable. But a discussion where users feel comfortable 
about asking questions and evaluator feels comfortable about giving 
answers — even though they may be tentative— exemplifies the proper 
atmpsphere. 

Second, the gist of the report, the phone call, the discussion must 
be based on sound evaluation practice and tempered by the users 'needs 
for and orientation to different kinds of Information. 

As you consider how to organize to ensure sufficient and useful 
^valuator-user interaction, then, you must jtake into consideration the 
/ evaluator's role and views about use, and the usersV questions, needs, 
* and expectations. You must also remember that a one-^shot interaction 

probably will not promote use. The dialogue should be an ongoing 

" ■ . ' ' " ■ , ■ ' - . " '- '■ '.- - ^ ' • ■ ' 

effort. For Instance, you or another user might read a draft interim 

' "• ' ' - ■ ■ ■- ~ ' ■ 

report and then suggest ways in which the evaluator could develop graphic 



~^atar^TspTau^for 

. ■ . . ' ~ ' • ■ ' ■' ^ ' ' 
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time with you or other users analyzing findings, interplreting their 
meaning, and developing realistic recommendations. \ 

— INSERT 28 HERE — 

Another category deals with the substance or nature W the vattia- 
tion informacion which is reported. ; 

Two ^aluafioii substance factors have been shown to influence use. 
They are: ; i 

.•f~. . • • ' . • 

Information relevance 

Information specificity 
In some respects, organizing for these factors means" making sure, that the 
general considerations related to quantity and quality are kept ih mind. 

- INSERT 29 HERE - 

iMic tion can be regarded fas^^ relevant if It is pertinent to users, 
if it bears directly on their issues of cohcern, if it is to the point. 
These characteristics must be Judged from the point of view of the 
intended user(s) . F^^^ given user, relevance will often be a function 
of his views about the project, the questions he has, and the urgency he 
attaches to them. Therefore, an oncaing dial orue is critical in helping 
the evalivator to understand the nature of user questions and to address 
them pjertinently and directly. . 

There are seyeraVpotejitlal trouM^^ 1 . ^ too many 

quest ; ^.i^ ... a asked of the eval uation , 1 1 may be dlf f icul t to col 1 ect and 




provide relevant Information on all of them. Perhaps the number of 
questions can be reduced. If not, you shquVd be aware that the 
evaluatipn win probably provide, only brp^'d and general information. 

Second, the evaluation questions may be too narrowly conceived — 
requiring only the most perfunctory of answers. .If potential users find 
such answers relevant^ well and good. But they will probably want 

something, more. In this situation, you may have to help reconceptualize 

. ■ ■ ■ '. ' ' . ■■ ^' 

and restate questions so that they require rnqre than routine responses. 

■ - ■ ' •■ / ' " ■ , ' ■ ■ ■ ■ 

' ' ■■. . : •■■ ,,. /■■ - - - 

Third, the evaluation questions may be too broadly conceived, so 
general that a variety of information types and answers could be deemed 
relevant. Again, such questions' will^robably need to be reconceptua- 
lized and rephrased. / 

In your organizing work, then, you may need to negotiate and modify 
the range of questions asked/of the evaluation: by reducing the number 
of questions, prioritizing questions; broadening the focus of ^n 
questions, and sharpening the focus of general questions. The point of 
this effort is to ^a(c6 sure that users* questions can be answered in a 
way that Ms relevant to them/ that the evaluator understands what kinds 
of informatiofr are relevant to what kinds of users, that users are aware 
of the priorities attached to various questions, and that the information 

/ ■ " ■ ; . . ' .■...■■./' ., ■■ . ' -, ■ , •: , ■ 

/ ■ - . ■ . * ■ - " : c ■ . . . ' : . 

provided has the potential for use. 

■ . • ••■ . : • ■ • ■ ' 

/ • ' ■ - . ■ . . ■ ■ • 

/ — INSERT 30 HERE — " ^ 

/ » . ■ • • . .1 . . ■ . 

/ ■ ". . ■ ■ ■ ■ ■ '•• , ■ ■■ . - .• 

Having decided what general features of relevance are important, you 
must "next think abolit what ^T^^^ kinds of^Jnfomati on should ^^^b^ 

^ .-. . • ■ ^ f - ; y .... _^ 
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provided to variousi users. Now, two different users, or user groups, may 
share a coOTon question and a conrnon Information need. But even so, they 
win look for different Information specific to their particular jobs and 
responsibilities. For example, users with operational responsibilities 
may need fine-grained Information to help them with their dally 
activities, whereas users with administrative responsibilities In several 
areas may need Information of a more general nature co of a comparative 
nature. 

The point Is clear. Questions that are unique to one user or user- 
group require an answer specific to that question. Questions that are 
shared by several different users or user-groups probably require 
evaluative Information at a level which differs for each group. 

Research shows that the more the information is specifically 
selected and focused toward the needs of a particular user, the higher 
the potential for U5e. Therefore, the specificity issue has the same 
kind of organfzing implications as were pointed^ut for relevance -- you 
may need to help the evaluator to understand the information specificity 
required by each of the potential users. 

. ' — INSERT 31 HERE ~ ; ^ 



He now consider a group of factors dealing with the reporting of 
evaluation information. ' 

Evaluation reporting' has been defined as consisting of five 
factors. They are: 

Frequency of information provided . ~ ^ 

Timi ng of i nf orma^i on 



Format of presentations 
ormat of report 
Statistical /narrative data 
Each of these factors contributes to the quality of ongoing dialogue. As 
we discuss them, keep in mind that, in your organizing effortSj, you 
should concerned equally with the technical quality of the report in 
addition to its appropriateness for the particular context or setting. 
Since the five factors are closely related, we will first describe each 
factor and then discuss the 1mpl1cat1or\s of ail five for your evaluation 
organizing. / ^ 

The frequency of ijaformatidn provided bears some relation to amount 
^ of interaction between evaluator and users* "Frequency" here, however, 
refers specifically to the extent to which written or verbal evaluation 
information is reported to users throughout the life of a project. 

This factor may have a strong bearing on use. Although the effect 
on use of a final, formal evaluation report is difficult to gauge, 
research suggests that timely and well -focused formative or progress 
reports incl^ease the potential for use. ' 

The timing of information — that is when it is provided, whether in 
written or oral form — can affect use potential. Reports that are 
issued at times which do not mesh with key project events and decision 
needs obviously cannot be used; for example, a report that arrives too 
late to help in making a certain decision. Further, the late submission 
of one report creates a mind-set among users, who may decide to Ignore 
subsequent reports, even if they arrive on time. 



In addition, well-timed progress pr interim reports, oral or 
writtp^ can provide partial answers to users'' questions oyer time, 
thereby increasing the likelihood th<. ; they itlclnatf' and apply 

subsequent information, and helping to convince users of the evaluator's 
overall credibility. 

While evaluation information can be presented in either oral or" 

written form, each type of presentation makes its own unique demands. To 

be effective, an oral presentation requires a balance of formal and 

informal delivery style. For example, at the. end of a formal prepared 

presentation of findings, the evaluator may informally amplify a point or 

two. He may ask for questions and respond informally. He may interrupt. 

a lengthy presentation at some logical point to opeih the discussion to 

his audience. Research shows this mix of the formal and the informal en- 

\ ^ ■ . ' ■ ' . ■ ^ ■ ■ 

hances the effectiveness of oral presentations and increases likelihood 

that the information it presents will be used. 

Like oral presentations, written presentations such as progress 
reports and final, reports require an appropriate format to be effective. 
The "format" of a written report refers to its visual features, indepen- • 
dent of its content. As with other factors already described, "appropri- 
ateness" is a function of user needs and preferences. 

Research shows that a balanced mix of graphics (tables, charts, 
figures), technical presentation, and nontechnical narrative can enhance 
the use potential of written reports. 

Evaluation information is usually presented in both statistical 
( tabul ar) and .narrati ve form . Some presentations, oral or written, 
overemphasize one to the neglect of -the-^other. Statistical -and iiarratfve 



data should complement each other as findings are presented and 
interpreted. 

We have found that when an evaluation report is overloaded with 
s data, the potential for use diminishes. But when such data 

are supplemeuu^ by rrai ve OK^josit' ^n (interpretation of findings, 
formulation- of reconroendatlons), the potential for use increases. F' 
when interpretation is not substantiated by statistical data, a repori, 
loses credibility and has a weakened potential for use. Conversely, 
interpretation and reconsnendations which are supported by appropriate 
data tend to increase. the use potential. 

Each of the five factors described above — frequency, timing, fof- 
mat of oral presentations, format of written reports and statistical/ ' 
narrative data — should bie-^onsidered in your organizing activities. 
For example, what «|oes "frequent" mean? For one potential user 1t may 
mean getting information at the beginning, of the project, at some' 
mld-polntj and toward the end. Another user may regard anything less - 
than a monthly updating as Infrequent. For some users, monthly 
communications may reach the point of overload and end up'be1ng'*f1lVd"-- 
and^lgnored as quickly as possible. Simply talking to users about this 
issue is probably the best way to determine frequency levels. You should 
talk to the evaluator -to be sure that he understands the inforination 
.frequency nee^s of various users. Bear in mind that his commitment to 
use and his orientation toward user involvement, along with the fiscal 
resources available to him, can have a bearing on how of teri; information 
is provided. - ^ 

o 

ERIC 



i. Timing can be planned In a variety of ways. Obviously, If you 
report to some external authority which expects reports- by a certain 
date, reports should be there by that date. On the other hand, it might 
be Bf^ter to send review drafts to these authorities In enough time to , 
receive their comments and feedback and to respond to them In the final 
report. If they are potential users of the Information, not just report 
.readers. Involving them In the preparation of the report may stimulate 
use. ^ - 

Other users, such as project staff and project and central office 
administrators, should be given the opi^ortunlty to help plan the t1m?ng 
Qf oral and wrltters presentations. For example, they shfluld be asked 
when they will need Information on a specific question so they can apply 
It to a decision or problem. If this advice can't be obtained, and if 
you know enough about the various users and their needs, you riiight work - 
with the evaluator to set timing by an^tlcipating when the Information Is 
needed and working back from that date to. provide sufficient lead time. 

Though we mentioned earlier that oral presentations are most, 
effective when they combine formal and Informal features, that 
generalization may be too broad. When you discuss this matJer with the 
evaluator, you should be guided by what you know about the preferences of 
the u^ers who will come to the presentation. If you feel that i formal 
Style Is more appropriate to the audience than an informal style,, or vice 

■ " ■ ^ 

versa, you Should niake that clear to the evaluator. 

"As to the need for a statistical and narrative mix In oral and 
written presentations, and for format variety in written reports, the 
only possible problem you are apt to encounter is that the evaluator will 
have a different opinion. An evaluator who takes a project-serving role. 



who is committed to use, and who values user involvement /will probably 
welcome your suggestions about balance in content and variety in format. 

In other cases, you may have to sell these ideas to the evaluator. 

■ ' . ' ' ■ ■ r 

Whether you should use a hard-sell or soft-sell approach depends on your 

authority over the evaluator* 

— INSERT 37 HERE — . . ' 

* * • 

The evaluation procedures that may -affect use are: 
Methods 

Handling of mandated tasks . 

Use of a general model ^ j ■ . . • 

~ INSERT 38 HERE — 

The methods selected for an evaluation may be considered both in 
terms of their appropriateness and their rigor* Appropriateness ~ 
the suitability. of a particular evaluation method' for a particular 
evaluation purpose — has two aspec.ts. 

The first is the question of methodological appropriateness. For 
example, an evaluation may plan to use a certain measure to collect data 
on a given point* The question here i?' whether the method as method Ms 
suitable. If the^pfoposed measure iV^n interview, then you must ask if 
an interview is the most appropriate means of answering the intended, 
question. If the proposed measure is a questionnaire or rating scale or 
paper-and-pencil test, the same test applies: Is the proposed measure . 
the most appropriate for the intended purpose? , » 



The second Is the question of context appropriateness. Is Uie. 
proposed measure appropriate in the particular project context? . For 
example, it may be that a given question can be answered equally well by 

0 > ■ , 

a questionnaire* or an interview. If one measure ^ems more ^ 



user-appropriate than the other in a given settiHg, then it should be :^ - 
used (unless fiscal constraints apply and cannot be offset -jn ^ome- way).,. 

When you organize for methodological rigor, your major concern is 
precision and accuracy. You want information that is as mi stake-free/as 
possible. Like appropriateness, rigor should be considered from a dual 
Standpoint, because qualities such as accuracy can be^ gauge^d by two 
yardsticks. - ^ - . 

First, there is the kind_of rigor expected of good evaluation 
practice. For example, rigor demands that, if a formal interviewjs to 
be cortriucted(|as part of an evaluation, minimal criteria must be applied 
to assure accuracy. Each interviewer must follow the same procedures. 
AIT interviewers should ask the core questions in the same way. If 
interviewers are to ^cap,ital1ze on opportunities to explore core responses 
In more detail, they should all do so, in the same way. This means the . 
interview guides should include a set of probe questions as well as spe- 
cific instructions on when and how to use them. Further, each Inter- 
viewer must record the responses in the' same way, and the method of data 
analysis^ must be suitable to the evaluation question and to thejnforma- 
tipn collected on it. • : 1 \ 

Second, beyond professional measures of rigor, there is user percep- 
tion of rigor. In judging rigor, users may ask whether the evaluator did 
what he said he would do, or whether a particular measure was used with 
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ng. To paraphrase a famous 

'J 

be done but must be seen to 



care. For example, the^valuation plan may call for hiring part-time 
staff to adminis.teK an Interview. Will users see this as rigorous admin- 
istration? What if the part-time staff consists of graduate students 
from the local university? IJsers^ may feelj that rigorous application 
demands that the interview be administered^ by the evaluator or by project 
staff sensitive to the institutional settii 
dictum about justice: Rigor must not only 
done* 

Research shows that ^appropriateness and rigor can have a bearing on 
use. Dialogue between the evaluator, you, and other users should help to 
assure consensus on what measures are most appropriate and rigorous in 
both a technical and a qontext sense. ' It is all tod easy to get blown 
out of the water because the data from -^ perfectly sound measure are 
challenged on the ground?; that the measurv was used in a way which was 
inappropriate to the setting* 

— INSERT 39 HERE.—' : 

The extent to which an evaluation deals with its mandated tasks, and 
the manner in. which it deals with them, may have a bearing on project- 
level use. As we saw, earlier, some mandates and requiremeptsi may be 
'imposed by agencies outside t;he project being eva\u^ted. Other obliga- 
tions may originate from within the project. If the evaluation over- 
emphasizes, or is perceived to overemj^hasize, externally mandated tasks, 
project use may be threatened. On the other hand, the externalr agency . 
may be one of the potentiaV evaluation users. If the agency believes 



' that the ev2(1uat1on is overly focused on project needs, then potential 
for external use may diminish. 

We have already noted that the evaluator who tries to "serve too 
many masters" may experience role conflicts, which serve no one very 
well . Again, youV organizing role is critical here. By setting up 
opportunities for you, the evaluator, and other users to share in an: 
early and continuing dialogue about how best to handle various mandated 
tasks, you aire helping to ensure that no/ one task or set of tasks is 
overemphasized. You are helping to ensure that the number and range of 
tasks can be managed and answered. You are nel ping to ensure that the 
people who set the tasks, and who want answers to their questions, will 
get ^information they can put to use. . ^ 

— INSERT 40 HERE — 

"Model," as-zused in d researc'n context, usually refers to beliefs 
about, philosophy governing. Intentions behind, manner of conducting, and 
the product of ^something such as evaluation. Generally, evaluators win 
structure activities differently dependent upon which model they sub- 
scribe to. there are various models that may be used to guide an evalua- 
tion. While It Is not necessary for you to know or learn about them^ the 
following represents a sample of the more prevalent models or approaches: 
CIPP, CSE model , Responsive Evaluation, Goal-free Evaluation, Adversary 
Evaluation, Connoiseur approach. Some evaluation models are unnamed and 
simply grow but of a belief that good evaluation .Is synonymous with good 

researiph, 

■ V- 



Our point here is not to advocate one evaluation model or approach 
over another. But the models mentioned (and many others) are out there, 
and the implication for you is quite clear. cOne eyaluator may prefer one 
approach over another. Indeed, he may insist on one approach over 
another and may try to use it in virtually all situations; you and your 
project may hot fit his predilection and this will possibly have 
disastrous consequences lor. eval uation use. 

You may get an evaluator who tries to convince you that one 
particular approach is necessarily "better" or "more accurate" or "more 
scientific" than another. Or you may get an evaluator who tells you that 
some formal model has to be followed. 

But, if you are concerned about the evaluation having Impact , the 
research literature doesn't support either position. Following a formal 
model does not necessarily ehhance use potential. In this sense, no 
model is inherently better than another; Use potential will. Increase, 
however, when the evaluation uses methods that. are technically 
appropriate and user-sensitive and when it offers the kind of informed 
balance we have spoken about in this chapter. In addition, the role that 
the evaluator chooses has a greater bearing on the use potential of his 
"approach" than any other single factor. ( In U sense, you may want to . 
consider these suggestions and the approach embedded in this book as a 
"user oriented model ' i 

In organizing for evaluation use along the lines we've suggested, 
you cannot al 1 ow the eval uattpn^ to be any thi ng 1 ess than a tool for 
satisfying the information needs of your Intended uSersv You should make 
sure that the evaluator has a thorough understanding of your definition 



of use, as well as, a dejtailed description of your users and their needs. 
Anything less than a hajTnonious match between the two can have a bad 
effect on the use potential t)f an evaluation," 



Here are some questions, drawn from the preceding factor descrip- 
tions, which you can refer to as you organize for use. 



INGS TO THINK ABOm 



Have you arrived at firm or tentative^-^answers to the questions 
asked at the end of theWctions on human and context factors? 

• - \ . ■ ■ ■ ■ ■ - ■ ■ . 

Do these answers suggest -^he forms that your evaluation should 
take to be used in your prioject setting? 

Will your users' interest ip the evaluation be sufficient to 
ensure a heal thy ongoi hg dial ogue with you and tne eval uator? 
' ■ ' ' \ - ' ^ , 

Are the users and their evaluation qifesttbns clearly delineated 
so that you and the eyal uator \can decide on how to provide 
relevant and specific information? 

' • ■ ~ ' ■ . • ' A' ■ ■ * ■■ 

What^kinds of evaluation procedures should the evaluation 
employ to^ensure that findings are credible to the various 
users you have in mind? 

How often and in what forms flight evaluation information be 
provided to meet users' decision needs? 



On the basis of your answers to these questions— and to other 
\ "... ' ■ • . . . • f 

questions you may raise yourself 7- which evaluation factors do you think 

' .■ ^ -, " 

you will have to Influence personally in order to promote use? With 

these evaluation factors in mind, do you have a clearer picture of all 

the human and context factors which will impinge on the manner and extent 

of your influence on the , evaluation? - ' 
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CHAPTER IV 
EVALUATION FACTORS: A' PATTERN FOR USE 

• The use-related factors introduced in the previous chapter are 
essential elements in organizing for more effective use of evaluation 
information. In this chapter we will provide you with a structure that 
' brings these factors together. -The structure, which we call a^ factor 
pattern , > serves two important functions: 

1. It illustrates how these factors can relate to each other to 
increase evaluation use; and 

2. ^ It gives you, the administrator, a guide in organizing your own 

evaluation for enhanced use. ' • ^ 

^ . 

This chapter will also offer you a hypothetical case study of an 
evaluation setting — a bilingual education program in "Hilltop" — 
within which you can begin to practice your organizing tasks, using both 
(the factor pattern and^sbme of the factor^related tapctics we discussed in 
Chapter III. ; , . ' 

Why have a factor pattern? ' ' 

We have designed the factor pattern as the hiost efficient means of syn 
thesizing what we know about evaluation use. It reminds us, first, of the 
interdependence of factors in contributing to evaluation use. No factor 
Stands alone as a completely independent element; the role of each^factor 
is conditioned by its effect on other factors and the effects of these on 
-it. Think of the, factorv"evaluator rapport" as just oni example. It ' 
Should be apparent that even where rapport may contribute to your intended 
use of the eveluaUon, such as direct action by teachers, rapport is more " 
lii^ely to make a difference if there is also. say, a certain lisvel of 



teacher interest in the evaluation or if the information provided is in 
a form the teachers can user In the same way, teachers^* interest in 
the evaluation, by itself, may mean little tf the information provided 
is inadequate or if the e^^aluator actively alienates the teachers. The 
factor pattern indicates which factors are likely to influence each 
other most directly in bringing about intended uses of the evaluation. 

Second, the factor "pattern places related groups of factors in a 
Topical order . The pattern As a graphic representation of the simple 
fact that in organizing for evaluation use you will have to start some- 
where. And where you start is crucial. By dealing with certain factors 
first, you begin to define and narrow the scope of your organizing task 
for the factors that follow. Looking again at the example of "evaluator 
rapport," it would probably be a waste of your time to worry about this 
factor before you knew the characteristics of the users that the evalua- 
tor^ will be interacting wUh. Nor, logically, would you make rapport 
your last item of concern, especially where the intended users of the 
evaluatijn^e.g. , teachers in, the above example) wouTd te working with 
the evaluator over the course of an entire year. The order of factors 

within our pattern, then,, is more tltian^aus^t a handy, step«by-step map in 

\ * ' 

organizing for evaluation use. Order is also one of the ways, in which 

- < ■ • .■ ■ ' • ■ " .,- 

factors influence each other within the context of an evaluation. 

Finally, the factor pattern is applicabl^xto most evaloiation 
situations . It^is broad enough to account for the somewhat different set 
of factors that contribyt:e to potential use :for eviary combination of use 
and user within your organizatiioTi. To continue with our example,^ 



"evaluator rapport" may be important in getting teachers lo apply infor- 
-mation frorn the evaluation, but ^he, evaluator 's^contact with the state / 

^ r • ' . ^- . i : i ' / ' .-c. • . ^ 

and federal authorities 'may be too -limtted -for^r^pport to^ ^ much part 
in the direct action they may take. "Because each combination of user 
and' use is^ unique in this way, we have constructed a general factor : 
pattern that can be inodified^ according tofany of the use and user combina- 
rluns you choose to focus" onT^' thin your organization.. 
Understanding the FactbVPattern ' 

lie can better understand hew the factor pattern (Diagram 1 ) works by 
expressing it graphically. This will prepare you to work out variations 

^ » • // , ... 

of the patterns/to fit y own situation.^ It will also help you under- 
stand. hoV/; these variations can be applied in a hypothetical evaluation 
situatipn, (/ater in this chapter) as well ,as in your own efforts 'to - 
organize for greater evaluation use (Chapter 5).^ 

The factor pattern is organized by first grouping individual factors 
iato. "clusters," set off by. boxes. ^ — — r " 




A^luster tells you, the organizer, tliat th^se^particular factors are' 
related to evaluation use in the same way, distinct from other-clusters 
in the* pattern, and that the;se factors should iej^exainlned^ together ah 



at the same time. 

r / 

T^e clusters are then joined, by vertical arrows, 



^1 



This indicates the\order in which you should consider the sets of factors, 
since each cluster influences the content of . the cluster beneath it. 

The double-stemmed arrow at the end of the last cluster in the pat- 
tern indicates that ydur organizing task has been completed. It points, 
we hope, to the intended use x)f the evaluation by the users you have 
identified. 

For convenience, we have labeled each of the clusters to provide 
some better indicatiori of the factors included with . each. The labels 
also increase your understanding of the logical sequence of clusters. 



! / 



"Setting the Stage" " ~ \ 

"Identifying/Organizing the Participants" 

"Operationalizing the Interactive Process" 
"Adding the Finishing Touches" 



Now let's look inside the clusters. (See Diagram 1) Here we want 
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. Diagraiji 1 

Factor Pattern For Evaluation Use 



A/Setting the Stage 

Pre-existing evaluation bounds ^ 

User identity • 

Progr am characteristics 

Intra-organizational features 
^ External features | 

B. Identifying /Organizing the Participants 

User interest in evaluation ; 

User commitment to use . • 

Evaluator characteristics 
•background/identity 

• dommitment to use > 

• willingness to involve user in evaluation 

• choice of role 
•political sensitivity 

" - — -'credibility — — - — - 

Evaluation procedures — plan 

User professional style(s) 

C. Operationalizing the Interactive Process 

Evaluationproceduf.es — execution 
Substance of . evaluation information \ 
Evaljjator commitment to use 
Information dialogue— formative 
User information processing preferences 

■4^ 

D. Adding the Fmlshing Touches 

Evaluation reporting 
Evaluator characteristics (selected) 
Information dialogue— sumrgative 
User commi tment to use 
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to point out two things aboLJt how the factors are presented within a 
cluster. First, a factor can either be cited alone (as a whole) or it . 
can be cited along with'certain of its component elements. (Refer to 
Appendix for a complete list of factors.) Where we list a factor such' 
as "pre-existing evaluation bounds," we are referring equally to all of^ 
that factor's elements :r "written requirements for the evaluation," ■ 
"other contractual obligations for the evaluation," and "fiscal ? 
constraints." In the same way, we list "program characteristics i" 
"user identity," .and most other factors in the pattern as a kind of 
shorthand for all of the elements that constitute each of those factors. 

r. ~ '■ ^ 

■ .i ■ 

Alternatively, a factor can be cited in a cluster along with only 
those of its elements that are mo^t likely to contribute to evaluation 
use at a given point in the organizing process. Where we specify a 
factor in this way, the elements are listed beneath the general factor 
and are set off by a dot. An example of this' is the factor "evaluator 
characteristics." When ypu begin to identify and organize the evalua- 
tion's participants, the evaluator's* "background and identity" and his/ 
her "choice of role" (along with other elements shown in Diagram 1) wiT 
have a bearing on the planned uses of the evaluation. Toward the end 
of the evaluation process, however, as you organize the finishing touches 
for making use happen, you will need to concentrate on other * 



char^acteristics of the evaluator, such as his/her "understanding of t 
political considerations" and the "rapport" .that's been established. - At 
this point in the process, the evaluator/i:^ "choice of role" has already , 
helped to shape the finishing toucheis and the evalua tor's "background 
and idehtity" ay*e probably^ no longer of as much consequence for use. 

The second point about how factors are presented within a cluster is 
simply this: " each factor ^hoiald be weighed and planned for individually', 
on its own merits, and as it affects and is affected by other factors in 
the cluster. We've tried to express this dual focus in a graphic way in 
Diagram 1. By separating the factors withi/i a. cluster by solid horizontal 
lines, and yet including them together within the solid lines of the 
cl^uster as a whole, w^ illustrate that the factors enjoy a certain 
autonomy even as they act together toward evaluation use. 

As we have indicated above, each of the four factor clusters repre- 
sents a distinct phase, or stage, in the evaluation process. Let's now 
briefly look at what these. stages are. . 

Factor Cluster A, which we've termed "Setting the stage," comprises 
those factors which are generally regarded as "givens" of the particular 
s&tting within which the evaluation will take place. We include here 
the characteristics of the program and* some of the organizational con- 
straints. Although these factors are to some extent fixed, you — as 
organizer of the evaluation may nevertheless be able to influence- 

• t - . 

them in ^uch a way as to promote use. " 

V The first factor cluster has "set the stage" on which^the evaluation 
unfolds, the "context" factors which provide the framework within which 
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the evaluation is conducted. Factor Cluster B, in contrast,, concentrates 
more heavily on the human factors which play an important part in the 
evaluation use process. Thus, Factor Cluster B involves "identifying and 
organizing the evaluation's participants." We inc^e in^is cluster 
such fhings as the characteristics of the evaluator and the primary 
potential uses of the evaluation. ' ■ 

Factor Cluster C, "Operationalizing the interactive process,*' repre- 
sents a transition point in pur. organizing for evaluation use. Factor 
Clusters A and B refer mainly to the planning stage of the evaluation, to 
t those activities that are carried out prior to the actual conduct of the 
evaluation. Factor Clusters C and D deal mainly °with conducting and re-' ^ 
porting the evaluatiofl^itself . Here we begin to consider the period when 
the^ GvaTuatTorTp^^ actual ly take~pllceT~PTT^^ V 

were anticipating future actions; Factor Cluster C reminds us^ that it is 
time for carrying out these planned actions. Here is where we'll put 
into practice the steps we'Vi planned, guided by the examination or Factor 
Clusters A -and B. If the planning job has been done well . then the evalua- 
tion should proceed smoothly; if not, it's still not too late to make some ^ 
mid-course adjustments. . ' / 

The activities suggested by the content of Factor Cluster C will, to ^ 
a large extent, determine the information available for use. Thus, the 
questions of what that information actually comprises and of how it gets 
transmitted to potential users become increasingly important to the issue • 
of evaluation use. , 

Factor Cluster D, "Adding the finishing touchesi" the last of the 
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factor clusters, represents i^e final phar*e in maximizing the potential, 
tor evaluation use. This factor cluster beloi^ns to that^point in the 
evaluation prbcess where most (if not all) of the information has been 
collected. In the earlier phasesv you considered those characteristics 
of the system and of users that might#ake a difference in evaluation 
impact. Now, In Factor Cluster D, tha fruits of, the evaluation process 
are displayed for all to see and to coiment -upon. 

Of course, there is still work to be done. Once all the j'nformation 
is available, it must still be communicated effectively to the right 
people, presumably the identified users. The users hopefully, will live 
up-to their designation and actually uie the information in some meaning- 
ful way. As you might expect from this brief rationale, the components 
of Factor Cluster D are more heavily weighted toward human" factors once 
agajn. .After all, people are the u^ers of any evaluation. Of course, 
you will have done your bes't throughout the course of the evaluation to 
increase the likelihood that evaluation will be "heard" — will con- 
tribute to the rgitionality of decision making. , \ 

Now that we've OLPETined'^the range of factors represented-b^^ 
factor clusters, let's apply the factor pattern in looking at organizing 
for evaluation use. In this way, we're sifre you'll be able to understand 
in a direct, practical way the important features of the pattern that 
we've outlined so far: "the interdependence of the factors, as well as 
their relative autonomy; the logical order in which the individual factors 
and the factor clusters are^ presented; and the flexibility of the pattern 
as a whole, which allows you to adjust it and apply it to your own 



organizational task. 

Hilltop . - . ■ .1 ^ ^ 

♦ For this initial exercise, we We constructdH a hypothetical case 
study of a school di^rict ("Hil-ltop") in which a federally funded 
IV^bilingual educational program (''SABER?) is due for a third-year^ evalua- 
^tion. After the brief description of the program and its cast of 'char- 
acterSi we'll point out some of the general' features of the case study 
that you'll want to keep in mind a^ you organize the "^i 11 top" situation 
for a more useful evaluation. We'll alsp outline some of the many 
evaluation-related topics of concern raised in the Hilltop context. It 
is around one of these selected questions th^t, together, we'll apply 
the factor pattern to the challenge of organizing ancl planning for 
greater evaluation lise. 

The Hilltop Case Study 

This is the second year of Hilltop district's federally 
funded btlingual education project, SABER (Student A^chievement 

through Bilingu al Education Resources), - In line with overall 

school district policy of the last few years, SABER emphasizes 
basic reading and math ^skills. The project's goal is to pro- 
^ vide a. smooth transition into an all-English curriculum. Six 
of Hilltop's 10 elementary schools participate in SABER: The 
project funded students are concentrated in grades K-4, 
reflecting the area's recetft and rapid influx of Mexican and/ 
Central American iimiigrants. 

_ . -91- 

K ,103 



The project in Hilltop was instituted in response .to 
Federal guidelines concerning non-English speaking enroll-^ 
ment figures. There was, a4^ least initially, no'^demahd 'for- 
.a bilingual project in Hilltop's small Mexican-American 
community or from the recently-arrived Spanish-speaking 
residents. Nor was there much support for a bilingual edu- 



cational project among members of Hilltop's board of educa- 
tion; several members, in fact, voiced doubts about the 
efficacy of u^ng Spanish language instruction as a basis 
for later learning English. During SABER*s first year, the, 
board's attitude was "wait and see." The Hilltop super- 
intendent was somewhat more supportive of SABER, but due 

more to the project's emphasis on basic sRills^than its use 

■ '■ ■ ^ 

of Spanish for instruction. Tne board and the superintendent 

r ■ ' 

are well aware, however, that Hilltop is in somewhat of a 
regional spotlight because of the district's rapid demo- 
graphic changes, with other districts watching to see how 
(and- how -wel lj Hi l l top deaU-wi th-™^^ ^ — 



SABER is operated centrally from the district office 

by Mrs. Mary Torres, the director. She supervises SABER in- 

struction at all six schools, arranges for periodijC^ 

service training for all Title VII teachers, and i^ the pro- 

♦ 

ject's chief decision-maker in matters of persbnnel and cur- 
riculum. Mrs. Torres also oversees the procedure for 
initial student identification and later transfer to 
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•regular school classrooms, even though these, procedures are 
handled on a day-to-day basis by Title VII teachers at each 
scho6l using project-wide surveys tests , and teacher assess- 
ment 'forms. 

, As director, Mrs. Torres organizes theyequired district 

bilingual advisory committee which, like its counterparts at 

each project site in the district, serves as an information 

channel to the Spanish-speaking community in Hilltop. Jhh 

committees exercise no real policy-making role, although 

most members do participate in bilingual classrooms as aides 

or occasional volunteers. This direct exposure.. to SA&ER, as 

well as Mrs. Torres' efforts to keep, the corrmittees informed 

<■ 1 _ ... 
of project progress and test results, has largely dispelled 

the concern of some parents with the idea of teaching in 

Spanish. Their first priority. is their children's acqCii- 

sition of English. 

On^y at the most heavily Spanish-speaking project 

■ " ^ \ 

school. Mason St." Elementary, where almost 85 perceritof 

the 630 students are in bilingual classrooms, has Mrs. 
Torres named a site-level SABER "coordinator" to ease some 
of her administrative and supervisory burden. Mr. Edward 
Lopez, Mason St.'s vice-prinbipal and feideral project coor- 
dinator, is an experienced reading and math teacher and has 
■ taught in bilingual projects outside Hilltop. He is re- 
sponsible for much of the implementation and supervision at 

■ ' . -93- 



the classroom level . But becaus^of his vite-principal ^. 
duties, Mr. Lopez depends on teacher reports, test scores, 
and other secondary sources of information in making most 

SABER-related decisions, e.g., student placement, main- 

■ . ■ ,1 

streaming, rhoice of materials/ 

The annual SABEjj^ evaluations are done by Dr. George . 

Johnson, the. Hi 11 top district's head of program research. 

Because of the federal project guidelines, the emphasis of 

the yearly reports, which are sent on to the federal 

authorities and to the Hilltop board of educatignj, is on 



the ^ s 
tests 



iudents' scores on national ly-normed achievement 



. But it is. Mrs. Torres who, in her role as the pro- 
ject./administrator,. uses these„proaect-wjde „a^ 
school test scores as a vital part of the decisions she . 
must make on teacher assignment, student placement, strenthen- 
]Jig weak classrooms, choosing appropriate in-service train- 
ing procedures, and subject area material s,^ expansion. of SABER 
into upper grades, and so forth* As. a. former teacher in bi- 
lingual projects, hpwe*er,,..Mrs: Torres recogn-fzes, the injf)or- 
tance of classroom activities not measured by standardized 
tests.\ As a result, she's' negotiated with\ Dr. Johnson for 
additional evaluation tasks ,Mncluding classroom observations, 
periodic reviews x)f the criterion-referenced tests and .locally-" 
produced curricular materials, and a'^umber of presentations 
and discussions of thfe evaluation findings with SABER'S teachers 



SABER'S first annual ^evaluation proved .to ^ a pleasant 



surprise to everyone involved. It showed S'^ABER stud^ents scor- 
ing large gains in math and Spanish reading and.smalleir but 
encouraging gains in English reading. The gains were greatest 
in thejower grades, where the superintendent and board had 
most emphasized Hilltop's back-to-basics policy, and were 
equally distributed among alVsix SABER schools. The- test 
results did much to. solidify^support within the district 
administration and in the community, and they contributed as 
well to acceptance of the project's native-language approach. 
Some teachers at Mason St. and elsewhere, however, still at- 
tribute the gains to the lower student-instructor ratio made 



possible by federally funded 
bilingual method itself. 

Both Mrs. Torres and Mr 



project aides rather than to the^ 



Lopez have become especially 



concerned with the results from the second year evaluation. 



Not only were the test score 



gains less tha'h' the preceding 



year, but teacher reports and monthly criterion-referenced 



tests confirmed a decline afjter the ^first year's strong per- 
formance. Their questions at this . point are: are these in- 
dications of a decline accurate? If so, what are the. possi- 
ble reasons (e.g., materials? teacher execution? native- - 
Janguage first approach?) Mrs. Torres already has some 
tentative answers, drawn from her own experience and 
occasional observations in the schools, but she is countilig 

: ■ : ■ ■ . ■ : ' ■ \ 
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cn the" evaluation to provide specific data. Looking ahead to 
t)ie consequencbr^of 'the coming third -year evaluation, Mrs. 
Tarres perceives some^project risk in that' the school' board • 
may decide not to continue the project- However ,\_personal 
risk for most key personnel , ^including Mrs- 'Torres, is mini- 
mized since they all have tenure within the district. Reas- 
signment forxMrs. Torres, though > would undoubtedly entail a 

loss of status and_ authority. For the teachers in the project, 

. ■ 

a lack of project continuance might well result ii^i some less 
desirable assignments. 

Mrs. Torres- £e4s a certain urgency in cbtairjing the evalu- 
ation data during this coming, third year of the project. . v 
First, she recognizes that the federal funding will be termi- 
nating at the end of .the year, and there will be the necessity 
for the school baard and district administration to make a 
decision on whether the pro jep-t will be continued a s a ^ 
district-funded activity. 5he is aware, that^; these decisions 
will need to be made in conjunction with the district budget 

• ■ ^ ■ 'X ■ ■ 

hear i ngs f or~-t4ie---f o^l owi ng-yeaiq^^J n^^^ nvol ve 

teacher reassignments ahd may potentially affect the hir,ing 
of hew personnel within the district. She also recognizes 
; the necess.it;y of making- the board aware of the upcoming ^ 
decision facin^them and of providing appropriate evaluative- 
information that might be used as a part of that decision. 
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Mrs. Torres alsq sees* the possibility that the evalua- ' 
tio_n data wj^^^^^^^^ other ways. She would hope 

that evaluation data would provide insights that she^might 
^find helpful in making project decisions about classroom 
materials or in-service strategies throughout the current 
year. That is, she sees the evaluative data as serving pro- 
gram improvement purposes throughout the coufs? of the up-^ 
comi^ng year, in addition to its potential for Jnfluencing 

/ decisions about subsequent funding. , • 

' '■ ■ 

Hilltop: An Overview ^ • ' * 

\, ' ■ ■ • 

The Hilltop case describes a project that^ enjoys a certain amount 

' • ■ ^ V 

of district support because of the appropriateness of its learning ob- 

\ \ \ " . ' 

jectives and the concern about edification of students from bilingual 



backgrounds. Moreover, it gains^some respect throughoLit the district by 
virtue of the positi.ve test scores at the end of the first year. How- 
ever, based upon t|ie second-^year ^aluation, there have been some 
questions raised about whether program gains are "real." Apparently, 
\the failure to attain the same large gains in test scopes as in the 
first year have led to some question about the impact of- the program. 
Moreover, these concerns have been fortified by teacher, report^; dur- 
ing thK^ear. Of conaern then, during this third year, is providing 
additionaNstudent outcome data to clarify the authenticity of pro- 
ject impacT^br^^ Furthermore, it is . important to gain some 
insights into clas^rodm<^^ ^ 
service-strategies, teacher cha^ct^i sties, etc.) 'that might 



contribute most to changes in student test scores. It ,is hoped 
tha^ this instructional process data could be used to further improve 
the project in the event that it is continued as a reguUr disti^ict > 
program. . 

I^n^^exploring the potential of the evaluation to. contri'bute to 
program improvements as well as to any ultimate decision on project 
continuation decisions, we should kee^^ mind three important, features' 
of the Hilltop situation. First and foremost iS the central role 
played by Mrs. Tprres , the project director. Her actions and influence 
reach every aspect of the project, and'as director, she also deals 
directly with every group or agency that has an interest in SABER, in- 
eluding Dr. Johnson, the evaluator. Remember that it was Mrs. Torres. 
who decided what additional evaluation tasks, beyond student testing, 
were to b.e conducted, and it is lW*Torres who will first receive the 
evaluation's findings. Not all the projectrrela,ted d are taken 

by the director alone, of course; nor can she completely control how 
.others per^ceive SABER. But Mrs. Torres is_ in the position of being able 
to provide evaluation-based information that others will take into 
account anii as such, she is clearly" the person who will be expectejd to 
organize for evaluation use. 



The second/ feature of the Hilltop case is the multiplicity of - 
, potential usei7S of the evaluation : the program director herself ; SABER 

and other teachers; Mr. Lopez, the program's Mason St. site coordinator; 
,,j:r,the district's superintendent and board of education; the Parents 

Advisory Council; the Hilltop community at large; and the state and 



federal authorities who fund the project, bach is at a different 
level of authority or influence vis-a-vis SABER, and each will find 

. - ' ■ • • . » 

uses for the -evaluation that reflect that position.^ The same 

■ ' ' «. ' 

evaluation information— student test scores, for example — might 

be used by teachers to re-group students in SABER classrooms and by 

the Hilltop school board to.decide on continued funding of the project. 

Likewise, ^'some u-sers will be concerned with a'wide range of evaluation 

topics while other users will limit their v^oncern to a. f^w-speei-fie — — 

topics. By way of example; * contrast Mrs. Torres, who deals with every 

aspect of 'SABER, and the community's bilingual^ adyisotT^nplttees^ 

whose role is primarily one"* of^publ^^ 

jrji^B_th2!:ijEeatu¥e-we^^^^ built into the Hilltop case is the 

various kinds of uses t'hat each user group can make of the evaluation. 
Mrs. Torres J as project administrator, could herself use the information 
in her choice of curriculum materials; This is an example of an action 
use. Another action use of the evaluation would be t^e Board's consi- 
deration of the evaluation findings and recommendations when it decides 
on project continuation. Action use may also be. made by a project mem- 
ber who may hold a lesser administrative or non-administrative role. 
For example, site coordinator Lopez might use the information provided 
by Mrs. Torres to better understand the strengths and weaknesses of the 
teachers and thus, to b^ better able to supervise them effectively. 

Mrs. Torres' efforts to change Jdeas or attitudes about SABER, in 
contrast;! are what we. have called conceptual use (e.g., convincing 
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Hilltop parents and other districts that SABER 's native language approach 

\ is effective). Another conceptual use occurs when the parent advisory . 

coiranittees are provided with, information they can use to consolidate 

^- support for the project in the cornmunity. Note'again that Mrs. Torre^s, . 

as organizer of the evaluation, is the central figure. 

Organizing for Evaluation Use 

As she plans, then, for organizing the Hilltop evaluation to maxi- 

mize the potential for use, Mrs. Torres will need to attend simultaneously 

to several evaluation topics as well as the potential user groups interested 

in each of those topics. For example, here is a sample of several topics 

relevant to the SABER project evaluation: 

0 What accounts for the decline in student^hieyement, and 
what modifications,, if any, should be made^ project 
operation? . 

0 How should Mrs. Torres allocate her t^me as administrator 
to best increase the likelihood of program improvement? 

0 Non-biiingual project teachers have doubts about the 
appropriateness and the efficacy of SABER'S native 
language first approach. y 

0 If SABER is to be expanded, recommendations will need to be 
made about staffing, grade levels to be included, project 
: — — 0pef^4Gn-§ii-T3el-ifies-,- curricul um, l^udget, and the 1 i ke . 




We have reviewed these topics" and selecte^d one for our detailed case I ^ 
^ study. The exercise focuses on the actions t^ be taken by Mrs. Torres to| 
maximize evaluation use. Now, consider the s'ituation Mrs. lorries is 



facing. It's the summer 'before the third year of the project, and she's| 
considering the evaluation to be conducted auring the next school year, f 



•100- 



112 



She has identified a prime evaluation topic -- namely, the decision the ' 
board will have to make about whether they will fund the program out of . 
district monies after this, the third year of operation under federal 
funds. She has also identified the school board as the potential . • 
evaluation user in this context. In addition, Mrs. Torres herself 
intends to use the evaluation for purposes of p-ogram improvement. She 
is concerned, therefore, about being sufficiently attentive to those 
factors which can maximize the impact of , the .third-year evaluation, with . 
the ultimate goal of helping to make the evaluation information more 
influential to the decisionmaking, process. 

As Mrs. Torres is deciding how best to proceed, she turns to Factor 
Cluster A as the most logical starting point, since she needs, in 
'essence, to "set the stage" for the evaluation. As Diagram 1 previously 
reminds us, the factors within this cluster are: pre-existing evaluation 
bounds, user identity, program characteristics, intra-organizational ^ 
features, and external features. If you were in Mrs. Torres' place, you 
would likely spend a 'great deal of time and effort examining each of 
the se factors in detail as it relates to this evaluation situation. How- ^ 
ever, we don't want to either overwhelm you or bore you with an ; 
exhaustive critique. Thus, in our discussion we will take a more, global ^ 
view of the elements within each of the factor clusters and consider, 
only some of the kinds of things, that Mrs. Torres might do at each stage 

of the process. ; , 

In the pages which/fallow, we'll phrase our discussion in terms of 
the problems that Mrs, Torres faces during the course of th^ year, and 



the procedures she follows in dealing with these problems. To help you 
better relate the discussion to the factor clusters, weHl provide a 
reference point in the left margin. 



/ 
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SETTING THE STAGE 



Pre-existing 

evaluation 

bounds 



Problem 

Because this is a federally-funded project, Mrs. Torres 
.is knowledgeable about her obligations to satisfy the federal 
regulations regarding evaluation. One of these regulations 
is the reporting of norm-referenced test scores. Mrs. Torres 
is well aware that there is some. discrepancy between the 
firsir and second-year test spore results; in fact, the federal 
program monitor wasted little time getting on the phone to 
her after receiving the second-year ev^uation report. 
Resolving this apparent discrepancy is, therefore, one of the . 
pre-existing bounds on the evaluation — not just because Mrs. Torres 
a-^n d the federal monitor are concerned about this aspect, 
but also, because Mrs. Torres knows that the board will be ask- 
ing about it later. 
Possible Actions 

"How would you organize the evaluation in order to respond 
to this pre-existing constraint? 



Organizing for Evaluation Use j 

Mrs. Torres decides that she needs to make sure that the 
evaluator understand^ exactly what reporting requir^ents must 
be satisfied in terms of .the federal mandates, as well as 

■ ' . ■ 'n •■ ■■ i • ' ■ . ■ ■ ■• . . 
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what procedures are necessary to carry out the norm- referenced 
testing. She wants to make certain that this issue is 
specified in the letter of agreement With Dr. Johnson outlining 
his tasks in conducting the^ evaluatioii for the dtftrict. If 
Johnson were an evaluator from outside the district, she would 
make certain that thase issues were clarified in the evalyator' 
contract; since Johnson is a member of Hill^top's Research and 
Evaluation Staff, there is no formal contract. 
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SETTING THE STAGE 

Problem • « 

The present Hilltop Board of Education is comprised of 
five members, elected at large from the community. Of these 
five members, thiree are "old timers" and two are new. Mrs. 
Torres knows that of the three "old" members, tvio of them are \ 
statistically-oriented and have in the past closely scrutinized 
the standardized test results. The third "old" member Js 
distrustful of numbers and statistics, and reacts negatively 
to having to examine the test results as they were previously 
reported. The two new members are fairly unknown quantities 
in terms of their attitudes toward statistical data, but Mrs. Torres 
does know that they are inexperienced in educational 
matters and know little f.bout the district's programs — parti- 
^cularly SABER.* ' 

Possible Actions 

How can Mrs. Torres plan to respond effectively to the range 
of users represented by the curreat Board members? 

- ■■ ■ " . . \ " 

Organizing for Evaluation Use 

Mrs. Torres decided on a two-pronged approach to this problem. 

■ ■ - . ■ t^*^' ' . • . ' ■ . - • 

First, she decided that the evaluation report should !;ave several 
di f ferent features ttian in the past : for exampl e, there shoul d be 



an appendix containing technical statistical d|ita\whjch will 

\ be targeted ^to the two "old" board members who ^ are likely to 

' ^ ' ■ ' /' ^ ^ ' .- 

want such data available; there should be an executive summary ° 

which presents in narrative form the s,ame informa^-- 

tion in terms that the non-statistically oriented bo%rd members 

i ^ 

\ can interpret,. Second, she realizes that she will need to try 
wto obtain additional information about the two new board members, 
plus she will need to provide information to them about; the 
project, its goals and its operating procedures. 7^ 

Mrs. Torres confers first with Dr. Johnson, to alert him to 
the need for preparing both the statistical appen^dix and an 
executive summary as part of his evaluation report. Nexfc^ she 
sends information about the project to the new ^oard members and 

invites them to personally visit and observe in? SABER classroom^. 

ll 

Finally, Mrs. Torres tries to identify any other personnel with % 

// 

whom she is acquainted who may be able to provide some insight . 

• ^ Jl- ^ 
into the two new board members —their likes, dislikes, views on 

education, etc. 




«5> 
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SEHING THE STAGE 



Program: 
characteristics 



innovativeness 



Probi em ■ - ' 

Mrs; Torres is concerned that the board might not 
fully understand the innovativeness of the SABER program 
and,' in particu|.ar, the nature and rationale of the native- 
language-first approach. Thus, she recognizes the possibility 
that the district might elect to provide program funding 
in. the name of SABER but based on a different concept— 
e.g., teaching English first. In that event, she might 
have to make the decision of whether to personally support, 
funding a program which does not emplby the native language 
approach, or whether she is willing to head another kind 
of program in order to both provide at least some special 
program to serve bilingual students and maintain her own 
position in the district. , 



Possiblev.Acti6ns " ^ 

What could Mrs. Torres do to help reduce the likelihood 
that the board will, fund another type of program rather than 
SABER as it presently is conceived? 



Organizing for Evaluation Use 

What Mrs. Torres needs to do is to gradually explain 
to the board over the course "of the .year the nature of the 
program so that the decision at the end of the year does 
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not subvert the original intent of /the program. She hopes 



/ 

/ 

that the board by that time will /understand the*' rationale 
behind making SABER a native-language-first program. Torres 
also has to clarify that rationale and approach to Dr. Johnson 
so that the -evaluation will be attentive to trying to assess 
the influence of the native-language-first approach/ 
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Int raorgani zati opal 
features: . 

other kinds ^ 
of information 



Probl em 



SETTING^^T^ 



Mrs. Tories notes that there are many :Sources of 
"behind the scenes" infcrmat^ion which might play a role in 

•the board's decision.* She feels that some of these informa- 
tion sources could be recognized and incorporated more 
foniiaTly into the\ valuation. She believes that attention 
to these other information sources would minimize the'possibility 
of the board's ignoring^he data provided by the evaluator 
in preference to some other, unattfibuted data.whicri, they 
maintain, is contradictory^ Forvexample, Mrs. Torres is 
aware thit the board has in the past been very responsive . 
to parent concerns and -opinions. She remembers attending a 
board meeting when a favorable evaluation report of. a 
district curriculum was presented, but the b^ard, citing 
unspecified unfavorable parent opinions of the curriculum, 

^ asked the curriculum staff to do major revisions. ' 

Possible Actions ° 

What coul-d Mrs . Torres do fi rst tV^i^^ fy the .other 
sources of information that might; be reaching the board and 
second, to incorporate these sources within the framework 
of the evaluation? 
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O rganizing for Evaluation Use 

Mrs. Torres needed^more input for this problem,' so 

she. set up a meeting with Dr. George Johnson to see what 

i ■ - ' .-' 

he migfit suggest.* They jointly agreed that it might be 

1/ " <• * ' ■ 

valuable to provide some indications of parent attitude * ^ 

toward SABER." Johnson in turn developed ^^questionnaire 

to assess how parents felt about their child's progress 

in the program, whether parents wanted their children to 

continue to have the opportunity to participate in SABER • 

the following year, and whether parents saw. some other > v 

.'types of impact from the program (e'.g., 'increased or improved 

coirmuni cations b.etween parents ^and, children, etc.) .. : ' 

In addition. Dr. Johnson agreed to keep his "eaV to 

"the ground" in order to pick up on any other information 

sources relevant to the board* s decision process which 

might be incorporated into the evaluation. Mrs. Torres 

would also be on the alert to identify such extr^-dvaluation 

sources. ^ - • - ■ ^ 

/ - • 
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SEHING THE STAGE 



Probi em 

Mrs. Torres also recognizes that parents are qot the 
Intraorganizational ' \ \ 

features: ^ only information source within the community. Many^|imes 

other kinds people comment either favorably or unfavorably about the ^ 

of information , 

program who are not connected with it. It is Important to 
recognize that they too may be a source of other information 
to the board. For example, several of the school board 
members belong to the Rotary Club and to the local downtown 
merchants association. There are always a number of people 
who would ask: "What the heck are you guys doing at our 
schools? , Having the kids walking around 
jabbering i n Spanish? Why don't you make them into real 
Americans? What kind of schools are you running? Are they 
just.playgrounds .where kids speaking other languages avoid 
Shaving to really learn?" 

Possible' Actions , ^ 

• What can Mrs^» Torres do -to counter the potential 
negative effects of uninfofmed public comments and alternately,' 
to capitalize on positive public opinion? 



Organizing for Evaluation Use - 

-Mrs. Torres wants to make sure that potential uninformed 
information does riot impede on the decision process and 
lessen the impact of the evaluation results* Thus, she" 
recognizes that it is important to mount a public relations 
effort which can provide an explanation to the community at 
large as to what the program is doing. She also feels that 
a\ is important to describe' and justify the conceptual .basis 
for this kind of program. She writes an article on this 
topic for the local community newspaper, arranges to be 
interviewed by a local reporter, an~a^talks to board members 
about the possibility of making a presentation at the 
Rotary Club meeting. She indicates in h^^^^ 
that she will not be discussing the results of the evaluation, 
but she points out that the program is being evaluated. She 
also 'describes the native language approach to Instruction 
and why it has been adopted for SABER. She also emphasizes 
that people are not being asked to accept this program on^ 
faith. She mentions the concepti^l basis for such a program 
and indicates that the board is open-minded about the issue 
of whether or not ^it is working satisfactorily in the Hilltop 
school district. She confirms that the evaluation is bfeing _ 
conducted by. 'Dr. George Johnson, an expert from the district's 
Research and Evaluation office, who will make results availabl 
to the board in time for them to make project decisions. 



IDENTIFYING/ORGANIZING THE PARTICIPANTS 

Problem ' 

The major decision facing the project in the coming 
year is whether it will be funded by the district. Mrs. Torres 
recognizes this issue as the most important question the 
evaluation can help answer. . Nonetheless, even though there 
is already a guiding framework for the evaluation in terms 
of looking at the academic achievements of students and 
certain other pre-specified outcome measures, there may be 
other concerns that the board members have expressed about 
the program or its operation that the evaluation can also 
help answer. 

What has been particularly informative to Mrs. Torres 
in regard\to identifying what some of -these other concerns 
or questions might be have been her observations of school 
board meetings and her informal- contacts with various 
•..board- members..:..!.She.,.ffiel.s:4,h,d,,sie.,has 
of conments' relating' to other issues beyond that of the test 
scores that she feels the evaluation should respond to, 
both in terms of strengthening the evaluation itself as a 
responsive activity and in terms of providing additional 
evidence that can ult||iiately contribute to the board's 
decisionmaking. ' ' 

■ ■ ■ ' ; ■ ■ r-- '' ■ ■ ' . 

O":'" "r ■ ' . ' ' ." ■ ■ . •■•/ •■' ■ ./ ... . ■ . 



Possible Actions . / 



other questions and. concerns have been identified ai;id will 
be dealt with during the evaluation? 



Organizing for Evaluation Use ' 

Mary Torres feels that it is necessary for fj'er to get 
together with George Johnson to consider the extent to which 
.the evaluation plan would seem to be responsive jto the 
questions and concerns either expressed or implied by the ^ 
board members. Johnson's plan had earlier been! submitted ' 
to her and had been approved; she in turn had sent an in- 
formation copy to the superintendent. She now/thinks that / 
it might be helpful to have Dr. Johnson make s|' presentation 
to the board, walking them thrt)ugh,.the 'evaluai|ion plan. 'Hrhf 
board would (thus have an opportunity to ^provi^de their input 
as to whether \the information they need to satisfy their 
individual interests is likely to be available through the 
evaluation as it is presently planned. Shejdecides to make 
contact with the superintendent to discuss %\\\% matter and 
to schedule a date for the presentation. 
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IDENTIFYING/ORGANIZING THE PARTICIPANTS 



ProbVern ■ ' " 

Dr. George Johnson has been in the research and evaluation 
unit of the school district for a good many years. Recently, 
most of his assigned work has tended to be. researc^^^^^ 
requiring him to develop and implement large seal e-data_col lec- 
tion schemes: Mrs. Torres is concerned about George's flexi- 
bility in this regard during the coming third-year SABER , 
evaluation. After all, the SABER project is not a controlled', 
classical experimental situation. Mrs, Torres is all too well 
aware that in the real world, data collection does not\lways 
occur as planned and on schedule. 

Ttiere is another characteristic about Dr. Johnson in his 
role as project evaluator which concerns Mrs. Torres. Sometimes, 
when George is engaged in informal conversation,' Mrs . Torres 
has observed that he doesn't know when to keep his mouth shut ^ 
when discussing potential'ly sensitive issues. For example, 
Mrs. Torres heard that George had been^sitting around the 
-teachers' coffee room the other night and, with a board member 
sitting at the next table, had been 'heard to remark that he felt 
the evaluation's results would indeed confirm a continuing decline 
in student achievement. Mrs.^rresrTioting that achievement , 
-testi ng had not yet even begun .worries that Johnson 's remarks 
may have been overheard by the board riiember and woul d thus 
reflect negatively in that person's mind in all future dealings ; 
with SABER. Mrs. Torres has already Ijeard from several teachers 
who were present in the room and'who were upset that George's 
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remarks reflected negative! y on their 
teaching. ^ 

1^htlne-^rres--eert^+nly-~respects-^-^J 

his ovn^ opinions, she also feels that he aeeds to know when not 

to express those opinions. In her view, Johnson lacks political 
. sexisiti vl ty ta .t surrQundlng. SAB^ 

seem to realize that as the evaluator of the project, his expressed 
-personal views may be misinterpreted as evaluation evidence 

by casual listeners. • 

Possible Actions 

What mi«ght Mrs. Torres do to encourage George to take a . 
more flexible attitude, towards data collection under SABER'S 
eyaluation plan and to let George know that he needs .to develop^ • 
npre sensitivity to political considerations when expressing 
his personal 'opinions? 



Organizing for Evaluation Use ' 

It seems clear to Mrs. Torres th.at she needs to have a 
serious talk with. Dr. Johnson about his evaluation, role within 
the project. There will be^ several aspects to this discussion. 
At the more general level, she wants to reiterate to him the 
importance of the way he conducts the evaluation. She will 
point out that ishe knows how concerned they both are about the 

, - " .V" • ■ ■■ ■ ■ • 

role the evaluation will play and their: mutual desire to see 

.■' ■...'<.' 

the evaluation findings reflected in the decisions to-be made 
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by the board. She points out, however, that he needs to be 
willing to make adjustments to the way the data might be collected 
for SABER. For exampl e , some "6r~t]ir^aT:irflraT'l3e-TiTi ss i ny -dne-ttr- — 
some external circumstances. Rather than just simply accepting 
missing values, George needs to be attentive to finding, another 

way to tdlTect simil ar data to repl ace that w^^^^^^ mi ssl ng. 

On a more personal level, Mrs. Torres will also expresJ 
her concern to George about the way his of f-ther-cuff comments 
could potentially undercut the evaluation and its usefulness 
to the board. She again reiterates how important the evaluation 
is to SABER'S future and expresses the" hope that, in fairness 
to SABER and to all the people who^' are working hard to try to 
make it a success, no prOgram-related information be misinter- 
preted by staff or by any board member prior to the formal release 
of evaluation findings. She feels that as a result of their 
talk, George will be^more discreet in his informal conversations . 
when in public areas of the school. 
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OPERATIONALIZING THE INTERACtlVE PROCESS , 

Problem 

T^iary~To r re s ^TTware^Tli^ 
place for a couple of months now, but she ha,sn't yet had much 
direct feedback from Johnson. She would 11 to know what he " 
has-been observing and if there are any problems that need to 

^be' arddressed: After all r prog^^ primairy^ ' "" 7 

goal of formative evaluation! . 

In, addition, Mrs. Torres knows that it is important to 
make sura that the District Office is aware that. the evaluation 
•Is indeed In progress and that evaluation info rmatfon is being 
used to effect iinprovement in the current operation of the . 
project. 

Possi ble Actions 

What can Mrs.. Torres do to increase the flow of evaluation 

information both to, hersel f and to the District Office? 

■■ ' * ' ; * ' 

■ t 

Organizing for Evaluation Use ^ : . 

Torres suggests to Dr. Johnson that he might want tb^meet 
with her on a regular bisis to keep her informed about the results- 
of his observations and initial data collection. She also 
suggests that he migKt consider writing a brief "status report^' 
or even a photocopied newsletter for distribution every month. 
This monthly update might indicate what *s. happening — ^tentative 
f^results — and a description ^of the ^kinds of evaluation activities \ 
that are taking place. ^She pointed out to Johnson that she, 
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the project staff, and the top administrators to the district 
might receive the report, along with people who are on the 
Parents Councfl" Tofres'f ee I s tWrrffe^aTurYep^ort-s^TTI"- 
serve several purposes: it willj alert her to initial findings 
and areas of potential concern, and it will alert teachers and 
other proje;ct staff to initial findings so that they can make 
operational changes. The pfesehce of ^^^t^ formative evaluation 
reported to the district administration may^also help in 
developiijg a sensitivity to evaluation information and an 
anticipation of the suiranative results. 



OPERATIONALIZING THE INTERACTIVE PROCESS 



Problem 



Substance of 

evaluation 

information. 



Mrs, Torres has. been feeling good about the forthcoming 

// 

evaluation report. She has seen several board members around 
the district off ice, A and th^y have rcommen ted to her that they 
are looking fonvard/'/to seeing the evaluation report and feel 

^ : a 

that it will be quite helpful to them, in the^ decision making. 



1 

She thinks that t|ie substance* of the report will be relevant 



/, 

to the board's concerns, because she is aware of the amount of 

^ a 

attention that she and Dr/ Johnson have given to making sure that 
the information is "on target." However, she has recently had 

some' slight qualms about the way things are going. 

■ ■ v/ ' 

At last/;Week's staff meeting^ for example, the'topicof* ' 
the parent (Questionnaires came up and several teachers reported- 

that the returns from parents had been very low. The parent 

// 

■ // • ■ • . ■ ■ . - , / 

questionnjaire had been given to* children to take home. This 

./ ' . ' . ' 

system usually works fairly wel.l ; howeviar, this has notTbeen 

the case wjth the SABER questionnaire. Mary Torres knows that ^ 

Dr. Johnson has made a sincere effort to get participation! 

The/questionnaire, was printed in both Spanish and English; the 

cover letter seemed to her to be effectivte in eliciting coopera- 

/■ ' ' ' ... " ' 

tion. Mrs. Torres theorizes that the low rate or return may in 

'■ ' •* -~ ' . 

fact be due to reluctance on the part of foreign-bom parents 
to participate in activities that they view ifT some way as related 
to government. She is not quite sure. However, with the very 
low rate of return (only around 10%) and virtually no responses 
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on the open-ended questions, Torres wonders whether Dr. Johnson 
will be able to provide convincing data to the board with 

Tesp'Ct~to~i;tren:^trB5tiTjnnrf^^^ '\ — — 

f 

Possible Actions n 

, Will the evaluation data'being collected by Johnson on 
parent attitudes and effects of. the program on the parent-child 
relationships be sufficient? Would the board consider the data 
relevant when there is such a small Response and no in-depth 
comments? Khat can Torres do? ' 



Organizing for Evaluation Use , , 

Mary Torres discussed the situation with George Johnson. 
She indicated concern' and asked Johnson for suggestions* They 
jointly agreed that.it wasn't very likely that very many more 
responses would come in. George recognized the problem of the 
possible reticence of bilingual parents to. complete the questionnare 
and suggested that there was a need for personal contact. He 
noted; however,\that he had insufficient staff to conduct all ' ^ 
the interviews necessary. He inquired of Mrs. Torres whether the ■ 
teachers could do a.follow up with a brief phone call to each / 
of the parents. Torres rejected. this idea, indicating that she 
did not warit to bias the evaluation results with the partici- \ 
pation of the teachers i n the data collection _ (parti cularly \ 
when a part of the data collection calls for the; gathering of 
. anecdotal data). Dr. Johnson and Torres agreed thstHtSinght 



strengthen the data base if jQhnson were to conduct several 
home interviews of parents randomly selected and write them up 
as m^ini-case studies to, supple ment the ques tionnaire data- 



/ 



I? 



-nib- 



ERIC 



±34 



. , ' ■ ^ ADDING THE- FINISHING TOUCHES .. 

Mrs.. 'Torres has just received some evaluation^ infor-' 
mation from Dr. Johnson regarding the critertO;:referenced 

User conmitment testing the teachers have been conducting during the ' 

to us)e " . 

fall. These results seem to indicate that student 

* , - performance is continuing to decline. Mrs. Torres 

^ ■ is initially inclined to, doubt these results, and to • 

assume that there has been some mistake either on George's 

, part, or else in the construction of the tests themselves. - 

Mrs. Torres catches herbelf, though, in the act of seem- 

- ingfy try to ignore the evaluation results — certainly 

not a position a use-oriented administrator should exemplify! 

But still .\.? 

. Possible Actions 

Faced with this dilemma, what could Mrs. Torres do? , 



Orqanlzinci for Evaluation Use 

At 'first, Mrs. -Torres decided to conduct a quiet investi 
. \ gation i^ato whether the tests themselves could perhaps be the 

cause of invali-d evaluation results. She quickly found, how- 
. - ever, that the tests did in fact, refliect the curriculum.' 
After brief discussions with Gearge,. she confirmed that the 
results ha4 been, presented accurately. Mrs . TorrlS«i4ecided 



in the absence of a prima facie reason "to doubt the eval- 
uation results, she should accept them as valid and look 
elsewhere for-the-explan^ition-as-to- the lowered student 
achievement.' She realizes that she had been somewhat 
neglecting her supervisoif^y responsibilities as project 
administrator and decides to begin observing again , in 
the SABER classrooms and keeping in closer touoh with the . 
teachers. She also makes a special effprt to geto'n touch 
with parents who have in the past been willing to provide ^ ^' 
her feedback about how the program was affecting their 
children to get their perceptions of this year's program 
operation. What she finds, somewhat^ to her surprise, is 
that several of the teachers were not in fact fqlfowing 
the planned program of instruction. Thus, is it not 
surprising that student achievement J s lowered on the 
criterion-re/erence^^ests which ^re geared closely to the 
SABER* curriculum. Realizing that her initial negative reac- 
tions to George's findings were unwarranted, she invites 
George to attend a meeting of the project staff ^o- discuss 
the content areas covered by the next set of criterion-refer- 
enced tests. She also asks- George to assist her by visiting 
more often in those classrooms which she has Identified as not 
following the prescribed curriculum plan. As a result of this 
episode, Mrs. Torres reconfirms her cdmnitment to using the 
evaluation results in^her own supervisory activities. ^ 



" ADDING THE FINISHING TOUCHES 

* • . ■ ■ " ^ i ■ 

Probl em 

Dr. Johnson has completed his final evaluation report 
for SABER and has; distributed it to the board members. - Now 
it's the evening of the board me'etipg at which George is to 
formally present his report. So far things seem to have 
gone well; the board members have been listening attentively 

and there have been, no external interruptions during his- 

preservtation. Midway through George's summary, however, one 
of the new board members interrupts and says, "Wait a miriute. 
Dr. Johnson, I just don^t understand what all these test ^ 

figures mean thalt yoiiWe been talking, abo^ut." 

. . • i ■ ■ ' . ' ' . " 

Possible Actions 

r~ / 

How should George respond to this question? 



Organizing for Evaluation Use 

George had been pleased with the job he did on the evalua 
ti on report - he felt that it treated SABER and its accomplish 
ments fairly and accurately. He had taken pains to rpake sure 
that the report contained se\^eraT features desi gned to ease 
thereader through the material (for example, there was an 
executive summary at the beginning, and an appendix with the 
more technical data at theyCnd, and several charts and graphs 
/as appropriate v<ithin the( body of the^ t However, lie 



wanted to make sure that he would do the best job he could 
of representing the SABER program in front of the board, so 
he had requested a meeting with Mary Torres the week prior 
to his presentation to get her suggestions.' 

His meeting with Mrs,. Torres was designed tp be a 
strategy session to brainstorm effective presentation tech-, 
nicjues. One of the strategies\ they came up with was to / 
develop, but keep in reserve, alternate ways' of analyzing!* 
and presenting the» findings. Thus, when the questioil arose 
during the actual board meeting. Dr. Johnson was able to 
bring put several of the^ visuals he had prepared after his 
meeting with Mrs. Torres. These visuals had been designed 
specifically with the new board members in mind, anticipating 
the types of .questions they might be likely to ask. In devel- 
oping' these additional materials, George Wantied to be responsive 
to the different information needs of the various users and 
their differeing preferences for how they liked information 
presented. Kad questions been asked by, other more knowledgable 
board members requestingfurther evaluation, George would have 
been able to bring out a different set of charts which he had 
also prepared in anticipation of the actual event. 

Not only did the board members* receive additional explana- 
tion and information from Dr. Johnson/ but the very fact. that ' 
he was so well prepared to respond to th^m Increased the board's 
confidence in the evaluation and\its finjdings.r 



Conclusioa 

Organizing your evaluation for use around the factor clusters we have 
described has advantages. The patterns are linked by a realistic logic, a 
kind of logic that you have probably followed in other situations. In many 
ways, it correspondsVith the kind of systematic planned activity you would 
engage in to make any venture successful. No less with' an evaluation situation. 

Your initial concern is with "setting the stage" — structuring the acti- 
vity, deciding what you want to accomplish, and what resources and constraints 
will limit your actions. A subsequent concern lies with the intended proce- 
dures and the various actors. Your concern as a manager is being aware of 
each, how they operate and the ways that you can facilitate accomplishment 
of goals. , As an activity progresses, you become aware of actions that you 
can take during the operational izing of the process. And, finally, as 
management activities come to a close you want to be assured that the finishing 
touches are added — that all of your efforts are properly reflected in the 
end product. It is similarly applicable in most evaluation settings, yet 
sufficiently flexible that it can be adapted to meet particular needs. The 
evaluation clusters simply "translate" this kind' of management thinking into 
the specifics'of an evaluation situation. 

in this chapter you have viewed the ways in which Mary Torres has been 
able to respond to organizing for evaluation use. In the next chapter, you 
will have an ^opportunity to apply this factor pattern and its several clusters 
to a problem of your own choosing. . J 
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At this 



CHAPTER V 
ORGANIZING FOR EVALUATION USE: 
APPLICATION TO/YOUR PROJECT 



p'oint, if you We been fpTl owing our line of reasoning and the 



illustrative/examples and scenarios in the preceding four chapters, you should 
be ready j;o deal with your/om project's evaluation J * 

Ffrst, let's summa^ze the steps ,that have lea up to, this threshold. 
We began by defining^baslc concepts such as evaluation use , evaluation user , 
and/evaluation factor (Chapter II), NextV we identified and described^ the 

array of possible evaluation factors, pointing out jalong the way soma of the^ 

/ ' ■■• ' - . 

interrelationships among- the different factors (Chapter III). Finally, we 

identified^ factor pattern with four clusters, presented a scenario, and 

examined the factor pattern within the context of 

Now it's up to you to assume"^ once again your admin 

charge of organizing your own evaluation situation 

Remember that the goa:l of this iguide is to s 

an evaluation so as to maximize the likelihood of 



that scenario (Chapter IV). 
i strati ve role and to take 



how you how to organize 
use. We have emphasized 



that the more the various users are involved in the planning, execution, and 



that evaluation use, %s we 
zed that the more you are 



analysis of the evaluation, the more likely it is 

have defined it, will occur. We have also emphasi; 

. ■ ■ • ■ " ' . ■ • 1 . ■■■■.-^ ■ ■ 

aware of the particular factors present in the evaluation situation, the more 

. . • . I. - ." ^/r'- ' . . ■ 

likely it is that you will be able to exert a positive influence on these 
factors, to insure that they work for, rather than against, evaluation use. 



ERIC 



■n 5- 



140 



Organizing Your Evaluation 

With these important reminders in mind, let us turn next to'the task of 
organizing your own evaluation. To help you in this important task, we have 
provided a series of worksheets,' one for each of the l^actor patterns with 
which you should now be familiar. These worksheets are- designed to allow 
.you to note important information pertaining to your' evaluation situation 
and to list some positive actions you and others can take to influence evaluation 

" use. . ' 

-On the next page, -you will find a diagram of the four factor clusters 
reproduced again foV your reference. This is to remind you of "the order in 
which you need to consider the factors which are most likely to affect y^ur 
evaluation. ' ' .' v , 

Following the, diagram is a series of four worksheets, one for each of 
the four factor clusters. These four worksheets are followed by an enlargement 
of part of .Worksheet A, dealing with the first factor cluster. Please turn 
now to this practice Worksheet A (p. 126). • . . 
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Factor Pattern For Evaluation Use 



A. Setting the Stage 



Pre-existing evaluation bounds 
User identity 
Program characteristics 
Intra-organizational features 
External features 

B. Identifying/Organizing the Participants 

User interest in evaluation ~ 

User commitment to use ' ' 

Evaluator chiaracterisfics 
" • background /identity 

• commitment to use 

• willingness to involve user in evaluation 

• choice of role / 

• political sensitivity 

. >crbdibility • 

Evaluation procedures—plan . 
User professional style(s) 

^' ... ^ 

C. pperationalizing the Interactive Process 



E'l^luation procedures — execution 
Substance of evaluation information 
Evaluator commitment to use 
Information dialogue— formative 
User information processing preferences 

D; Adding the^ Finishing Ibuches 

Evaluiation reporting 
Evaluator characteristics (selected) 
Information dialogue—summative 
User commitment to use 



4^ 



Organizing For Evaluation Use 





Evaluation Topio^si. 



Factor Cluster A: 



Pre-existing evaluation bounds 



User identity 



; Program cliaracteristics 



Intra-orqanizationai features 



External features 



With respect to.my own program and the above topic, I need to keep the following in mind 



Factor Affecting Use: 



A-1. Pre-existing 
evaluation bounds 

(written requirements ;- 
otiier contractual obliga- 
tions; fiscal constraints) 



.A- . 
\ 

A-2XUser identity 
(range; organizational 
positibciis; profossional 
experience levels) 



Relevant Information: 



Things I Can Do To Influence Use: 



,1. 
2. 
3. 
4. 
5. 



1. 
2. 
3. 
4. 
5. 



1. 
2. 
3. 
4. 
5. 



2. 
3, 
4. 
5. 



Organizing For Evaluation iJse 

>feheSt A (Continued) 




Factor Affecting Use: 



A-3. Program 
characteristics 

(age; maturity; innova- 
tiveness; overlap with 
other programs) 



Relevant Information: 
2. 

-3; ^ 



4. 
5. 



Things I Can Do To Influence Use:^ 

/ 

2. 

3. - 



4. 

5. 



\ 



\ 



A-4. Intraorganizational 
features . 

(central/district office 
roles & relationships: 
institutional arrangements 
& autonomy; other likely 
kinds of information; " 
perceived institutional 
& financial risk) 



1. 
2. 
3. 
4. 
5. 



2. 
3. 
4. 
5. 



•i 



A-5. External features 
(community^limate & 
influence; rote of other 
agencies) 



1. 
2. 
3. 
4. 
5. 



1. 

2. 
3. 
4. 
5. 



Organizing For Evaluation Use 

^ 




Evaluation Topic: 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures — pip.?- 



User professional style(s) 



-» ^ 



With respect to my own program and the above topic, I need to keep the .following in mind 

Things I Can Do To Influence Use: 



Factor Affecting Use: 



B-t User interest in 
evaluation 

(views about program; 
.questions & concerns; 
predisposition to eval./ 
. ^valuator; need for eval.; 
expectations and risks 
for the eval.) 



Relevant Informatiort: 

1. 

2. 
3. 

5. 



1. 

2. 



3. 
4. 
5. 



B-2. User commitment 
to use \ 



1. 


1. 


2. 


2. 


3. 


3. 


4. 


4. 


5. ' ■ . 


5. 



Organizing For Evalu\ti( 




>St . B (Continued) 



Factor Att^^^^ Icings I Can Do To Influence Use: 



B-3- Evaluatof char- 
acteristics (selected) 

(background/identity; 
credibility; choice of role; 
willingness to involve 
user; commitnient to use; 
political sensitivity) 



1.- . \ 


1. 


2. X 


2. 


■3. . / \ 


3. 


4. \ 


4. 


5. 


\ 5. 



B-4. Evaluation 1 . 

procedures — plan 



2. - 2. 

3. , 3. 

4. • • / -4. 

5. • 5. 



B-5. User professional 1. 1. 
styie(s) 

(admfnistrative & organi- 
zational skills; initiative; 2. 2. 
openness to new ideas) • ' 

3. * 3. 

^. , ■ 4. 

5. " .5. 



Organizing For Evaluation Use 

Worksheet 




Evaluiatibn Topic: 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



-» 4 -> 



With respect to my own program and the above topic, I need to keep thelollowing in mtnd: 

Things I Can Do To Influence Use: 

1. 

2. . 

4, 
5. 



Factor Affecting Use: 



C«1. Evaluation 
procedures— execution 

(methods to be used; 
dealing with mandated 
ta sks) 



Relevant Information: 
2, 



3. 
4. 

5. 



■ > 



C-2. Substance of 
evalua|lon Information 

(specific information r 
selected; information \ 
r relevance to audience! 



1. 
2. 
3. 
4. 
5. 



- A 



1. 

2. 
3. 
4. 
5. 



Organizing For Evaluation Use 





Factor Affecting Use: Relevant Informajio 

1. 



C-3. Evaluator 
CG-nmitn u 



3. 

1- 
5. 



/ 



(Continued) 

Things I Can Do To Influence Use: 

1. 

2. 

3. . 
4. 

^5. 



0-4. {nformation 
diaiogue — formative 
(amount & quality of inter- 
action between evaluator 
and user(s)) 



1. 
2. 
3. 
4. 
5. 



1. 
2. 
3. 
4. 
5. 



C-5. Usetr information 
processfing preferences 



1. 
2. 
3. 



1. 





4. 






' 4. 




5. 






5. . 
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Orgahizing For Evaluation Use 



Factor Cluster D: 





Evaluation Topic: 



Evaluation reporting 



Evaluator characteristics (selected) 



Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the fol Sowing in mind 



em 



Factor Affectir^g Use: 



D-1. Evaluation 
reporting 

(frequency; timing; format 
of report; statistical/nar- 
rative data; format of 
^present ation) 



Relevant information: 

1. 

2. 

.3..__ 



4. 
5. 



Things I Can Do To Influence Use: 
i. 

.2. 

. ^ 

4. 

.5. 



D%2. Evaluator char- 
acteristics (selected) 
(commitment to use; 
political sensitivity) 



1. 

2. 
3. 
4. 
5. 



Organizing For Evaluation Use . 

•Worksheet 0 (continued,) 



Factor Affecting Use: Relevant Information: 

* 


Things,! Can Do To Influence Use: 

* , ^ ■ o" 


D-3. Information • 1 . 

dialogue — summative 

(arnount^ type and quality 

pf interaction between ^* ^ ' I 

evaluator and user(s)) 

3. . " , 


2. . • , '. ■ 
- 3. . . • • 


4. 






'5. — . . 

■ . . ' • ■■■■■ -'^ " 








D-4. User commitment 1. 
- to use 


1. 


[ 2. 

« 


- ; 2. ' . ^ 


• • . 3. - 


3. 


. 4. . , 


4. \. ■ . 


• • ■ ' ' 5. . - _ 

\ 


0 . • - ■ • ' r "* 


•p - . " 


• r ■ 
• . o _ ■ 

; , , o - ' 



Organizing For Evaluation Use 

Practoce WoteSie© 



Evaluation Topic: 



Factor Cluster A: 



Pre-existing evaluation bounds 



^ser Identity 



Program characteristics 



Intra-organizational features 



External features 



With respect to my own program and tlie above topic, I need to keep tine following in mind: 
Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



A-1. Pre-existing ^ 
evaluation bounds 
(written requirements; 
other contractual obliga- 
tions; fiscal constraints) 


1. . - 
2. 


. ^ 1- 
2. 
3. 


e 


4. 


'4. 


o 


5. 


5. 



A'2. User identity 
(range; organizational 
{positions; professional 
. experience levels) 



. 1. 
2, 

^' 3. 
4; 
5. 



ERIC 



1. 

2. 
3. 
4. 
5. 
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Take a look at .the first line of the worksheet, labeled "Evaluation 
topic." What you should do here is to identify a major area of concern or* a 
major question pertaining to your project that you want the evaluation to 
he\p deal with. Note that the evaluation topic should be phrased as a question 
to make it as specific as possible. Once you have identified the appropriate 
topic of your evaluation situation, write it .out on the space provided. Go 
ahead and write in the book — the practice worksheet is provided for just 
that purpose. As you can see, we have gond ahead and written out our evalua- 
tion topic in the example below: " ' . 



organizing For Evaluation use " I iTl^SS ^Sti^ 



Factor Cluster A: 



Drnw..i«tinn Dvaluatlon bounds 



Usar Identity 



Program ctwactensiica 



Inlra-organizalional Isatures 



External (calures 



^ ^ ^ ^ -» ^ -> 



Wth respect to my own program and the above topic, .1 needlo keep the following in mind: 

V ■ 

Underneath the evaluation topic is reproduced the particular factor 
cluster under consideration, in this case. Factor Cluster A. The factor 

cluster is followed by an arrow pointing to the right { — ■ >). 

This arrow indicates that, once you have completed your work with Factor 
Cluster A, you must proceed to Factor Cluster B, the next logical step in 
your efforts to organize the evaluation so as '^to promote evaluation use. \ 

Following the factor cluster, we've provided a reminder phrase to jog v 
your memory about your own program's characteristics as you com j on to 
fill out the worksheet. Underneath the; reminder phrase, yoii'll notice three 
major headings: 



FACTOR AFFECTING USE 



RELEVANT INFORMATION 



THINGS I CAN DO TO 
INFLUENCE USE 
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. The first column lists the individual factors, which were described 
in Chapter III. We have already filled in this information for you in 
abbreviated form. In the case of Worksheet A, the first' factor is identified 
as "Pre-existing evaluation bounds." If necessary, take a moment to look 
back and refresh your menK)ry about this factor., 

. In- the second column, "Relevant Infornnation," you are asked"^ to note 
your observations about each of ^ the factors listed in the first column, as 
the factor appl ies to your situation. Think now about the evaluation topic 
you've just identified; think about your program and its operation; think 
about what may be required in the way of evaluation information and about 
what you may want to require your evaluator^to do to satisfy either your 
own needs or those of the other users; and finally, think about the fiscal 
and other resources available for the. evaluation. 

After you have thought thesia questions through,, summarize your thoughts 
in the "Relevant Information" column. To help you organize your observations, 
we've provided a numbered list in the blank column. Use as many numbers, as 
you need to write down all the relevant information. We-ve gone ahead and 
provided our own completed example below: ^ ^ 



\ 
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Organizing For Evaluation Uso 

Pracisce WcrksSiieet A 



Evaluation Topic: iWL iiK .iwc^ 

01/ engirt, at 1 



Factor Cluster A: 



Pfc-cxisting evaluation bounds 



User idennty 



Pfogfam chnrncteristies 



InUa-Ofgantzotional fcaiufea 



External features 



With respect to my own program and the above topic. ! need to keep the following in. mind: 



Factor Affecting Use: 



A-1 Pre-exiftttng 
evaluation bounds 

(written requirements: 
ott)er contractual obliga- 
tioTks; fiscal constraints) 



Relevant Information: 

\ 



Things I Can Do To Influoncfe Use: 



1. dUflooiitJedi^aiU.aui'^ 1. 

£^}U«filM^ a^i* Aw U>altt 

4. / 4. 

6. ' 5. 



When you have filled out the second colunin|of the practice Worksheet A 
to your own satisfaction, turn to the third column, "Things I Can do to 
Influence Use." For this column, think of the specific steps you can take 
to help insure that this particular factor will affect evaluation use in a 
^positive way. Your intention in taking these steps is to clarify, explicate, 
or modify the factor either to enhance its potential favorable influence on • 
use or to mitigate, as far as possible, any potential negative effect it may 
have on use, / 

When you are ready, go ahead and ^f ill out this third column as it per- 
tains to your own program. Be sure to phrase your statements in terms of 
definite actions; use concrete verbs; list only things that you or someone 
^Ise can do. You'll find our completed example below: • 



Organizing For Evaluation Use 



Evaluation Topic: iuajn* Jjfi^)^ 



Factor Cluster A: 



PfO-€xistlnq evaluation bounds 



User identity 



Program chafncteristtcs 



Inlra-ofqanizationni (caiufcs 



Extornal loatures 



With respect to my own 



program and the above topic. I need to keep the following in mind: 



F«ctor Atfecting Use: 



A«l Pre-existing 
cvalunlion bounds 
(written requirements; 
other contractual obliga* 
tions: fiscal consuainis) 



Relevant Information: Thlngi.1 Can 5o To Inducnce Uso; 



4. 



4. 

6. 



You've just completed one cycle of the process involved in organizing 
for evaluation use. You will repeat this cycle — first noting relevant in- 
formation on each fictor and then deciding what actions you can take to . 
enhance the potential for use throughout all the worksheets. For additional 
practice, continue now to the second factor listed on Worksheet A, "User 

Identity." ' 

In filling out the remainder of the worksheet, follow the same procedure 
as you did for, Factor A-1. In case you'd like another example for reference, 
we've provided our own completed practice worksheet below: 
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Orgnnizing For Evalustlon Use ' 



Evaluation Topic: Ji^lMJtJioci^ 



t Factor Cluster A V 



Prc«cxisting cvaiuation bounds 



User Identity 



Program charnctoristics 



(ntra-orqanizadonnl features/ 



External features 



With respect to my own program and the above topic. I need to keep the following in rpind: 
Factor Affecting Use: Relevant Inforrnation: * 



Things I Can Do To Influence Use: 



A«t Pre-existing 
evaluation bounds 
(written requiremonls; 
other contractual obHga- 
lions; fiscal constraints} 



A-2. User Identity 
(range; organizational 
positiorts; pfofessk>naJ 
exporienco levels) 



1. ^ ficQAito(6dii£at^tMQxitoa^ 1. yCg)icJyuA/uJhL4:ijohMihtfjird^^ 

3 y JuMC.k^i^'-U^^^)^^^^ 



4. 

5. 



5. 



1. JvcJocioUKiii^Mit cWjioj^ 1.iit aUt*xiLJinjfl4u^jtfu^ 

,i;^eMa> ASA? 



4. 
5. 



4. 



S. 



Now that you've coinpleted Worksheet A, you are ready to go on to each . 
of the three remaining practice worksheets. We've provided one . 
worksheet for each of the four .factor clusters; To work with these model 
worksheets more effectively, yoii may want to reproduce them in smaller 
segments, as we did for your sample worksheet. Or, conversely .m^.- ^ want 
to enlarge them so that you can deal with the whole -factor pattern at ice. 
Or you may want to 11st and; consider the factor clusters, and your course 
of action in organizing for evaluation use, in some other way. For those 
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who are interested, a supplementary workbook is available containing individual 
worksheets and step-by-step instructidns (Alkin and Jacobson, 1983). You 
will need to decide for yourself how you want to proceed. 

The Next Step 

Regardless of how youchoose to proceed, you will need to deal with 
each of the factors o'n all four of the factor clusters in order to complete 
the full pattern of evaluation planning, execution, and use. Of coursei. 
some of you may be in a position to complete all of the worksheets now — 
but we suspect that such readers will be in the minority. Different readers 
will be at different stages of the evaluation process. Most will probably 
be in the early stages and so will feel comfortable filling out one or more 
Of the worksheets, but not all of them at this time. If this is the case ^ 
with you, then you should go back to these worksheets at the appropriate time 
and fill them out as needed. Don't forget to look back at the worksheets 
you completed earlier to make sure that you have carried out the actions listed 
in the third column! , 

You may be someone who is not currently participating in an evaluation. 
Perhaps you are reading this handbook simply to increase your understanding of 
evaluation. If this is the case, you may also benefit from the discussion. 
We urge you to reflect- on the factor clusters and 9onsider how they might 
apply in situations you have encountered in~th"e~;pa:stT— Consider also what 
you might do in the future to make yourself more effective as a project 
administrator in organizing for evaluation use. 

Or perhaps you are not an administrator at all. Perhaps you are an 
evaluator ,: reading this book to see what kinds of actions administrators may 
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take in organizing your evaluation. We hope that the material presented in 
this book has forced you to take a fresh look at the whole evaluation enter- 
prise, that it has given you some new ideas about how. to perform evaluations 
-that will make a difference. You will find that it is very gratifying to 
know that your evaluation efforts will have some payoff, that they will 
amount to more than a report gathering dust on a bookshelf. 

To those researchers or evaluation theorists who picked up this book 
Out of curiosity, we can say that we hope you have not been put off by the 
absence of technical language and of the scholarly apparatus of footnotes. 
We hope that you have found some ideas here that will inspire and inform your 
own future work. You may wish to look more closely at the references noted 
in Appendix A as you reflect further on our comments. 



APPENDIX A 

\ 

'\ 'RESEARCH/LITERATURE SYNTHESIS 
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SYNTHESIS OF THE LITERATURE ON EVALUATION USE 

Prepared by 
James Burry 

\ 

Based on a review of the literature on evaluation use, the following 
factors appear to have ii bearing on the degree to which evaluation informa- 
tion may be /used. The individual factors^are placed in three major cate- 
gories: Human Factors, Context Factors, and Evaluation Factors/ 

Human factors reflect evaluator and user characteristics with a strong 
influence on use. Included here are such factors as people's attitudes 

toward Md in the program and its eval uation, . thei r backgrounds 

and organizational positions, their professional styles. 

Context factors consist of the requirements and fiscal constraints 
facing the evaluation, $nd relationships between the program being evalu- 
ated and other segments of its broader organization and surrounding 
community. 

Evaluation factors refer to the actual conduct of the evaluation^ the 
procedures used in the conduct of the evaluation, and the quality of the 
.information it provides. 

In the synthesis, the information drawn from the literature is 
presented as follows: 

Column one — Factor. This column lists those factors identified as 
having a bearing on evaluation use.* 

Column two — Research literature. This column presents information 
drawn from empirical research, in educational and other settings, on the 
-matter of evaluation use. For each work identified, statements are drawn 
typifying the factor it elaborates and its association with use. ' 

Column three — Theoretical /conceptual literature. This column Iden- 
tifies works from the theoretical /conceptual literature which have also 
identified the factor and/or association in^'column tv/o. 
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A. Evaloator Cliaracten'stics 

1. Coimiitiiieiit-to use of 
evaluation .results 
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Hlkin, Dai]|lak, S White, m Utilization is 
enhanced wlien the evaluator actively "strives to 
facilitate land stimulate the use of infonnation." 
Appropriate stimulation tactics I'rill depend on 
audience brjientatfon |p. 245) and the rapport 
established. 

! 

Braskaip S 3rown, 1980. Throughout the evaluation 
process, the evaluator itust take an active role in 
fostering utilization. F,e or she cannot assume that 
the audience will act on information contained in a 
final report (p. 92). , 

i(enne(|y,i982. The evaluator coiitted to use of 
information enlist the support of an administra- 
' tor. in this endeavor. This administrator, in turn, 
mc^ attempt; to "enlist' the support of subordinates 
in the use of information. 

The tactics used to enlist support down'the system 
m^y lead toiattitudes' ranging frdineutrality to , 
acceptance,: to hostile, to anxiety lpp« 122-123). 

This factor is associated. with political under-' 
standing audience composition, and aduinistrator 
si;yle. 

King &Pechii}an,. 1982, The evaluator committed to 
user involvpt iist be aware of the existence of 
kinds of "non-use" and their causes (p. 55).' For 
example, non-use msiy be appropriate if the results do 



Lange & Speiss, 1971 
Alkin, 1975 
HolleyUee, 1978 
Connor, 1979 
Holley, 1979 
Alkin S Law, 1980 
Cooley, 1980 .' 
Apling Kennedy, 1981 
Hansen, Martin, & Oxford, 1981 
Holley & Totusek, 1981 
Stevenson, 1981 
Thompson,' 1981 

King, Thompson, i Pechman, 1982' 
Chalimsky, 1983 
Holley, 1983 
Kean, 1983 

Radzikowski i Seaver, 1983 ■ 
Dickey n.d. " 




I.A.I. cont. 



2. View of desirability 
of user involvement 
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not deserve ;tp be, used or i^en tlie organizational and 
political context previ?nt5 use. Mis-use, or 
inappropriate use, may, occur if a user nianipulates or 
othery/tse misapplies appropriate results. Another 
'kind of ils-use, inappropriate non-use, can happen if 
results are maliciously ignored, treated >1th . 
indifference, or not coiunicated to users. 

Reisner et al, 1982. An evaluator who believes that 
"evaluations are a waste of time unless they are used 
by someone" ("p. 59) and i^o wrks "to create positive 
[user] attitudes toward utilization" (p.60l enhances 
the potential for. utilization. ■ 



Stecher, Alkin, IFlesher, 1981. ■ Utilization can be 
enhanced by Involving users in both the conduct of 
the evaluation and in planning processes for Its 
utilization (p. 133). 

Alkio, Stecher, i( Geiger, 1982. ■ Four types of ' 
evaluation/services can be provided in the interest 
of fostering user involvement in the evaluation: • 
increasing user understanding of evaluation; 
improving user capabilities' in testing and data 
interpretation; providing consultation on how 
findings mgy be used; and helping develop evaluation , 
procedures to meet local needs. To the extentythese 
activities foster local involvement, the evaluator is 
better able to deal with such factors as audience . 
expectations for the evaluation, Uieir preferred 
forms of - information - evaluation and others, and . 
the most appropriate inner of reporting, thus 

building a joint concern' for utilization, (p.. 55) 



3. THEORETICAL/CONCEPTUAL 
LITERATURE' 



i J' 



Windle i Bates, \m ■ ■ 
Alkin,/i975 
Caplanl et'^al.., 1975 
Hol1eyVlee,"l978'. : ' . . 
Connor, 1979 . . :■ 
Holley, 1979 ' ^ ' 
Alkin Slaw, '1980 
■Cooley, 1980 

.Hansen, Martin, & Oxford, 1981 
Honeynotusek,1981 ^ 
^^St^venson, 1981, : . ■ ., . 
King, tapson, & Pechman, '1982 
Chelijusly', 1983 
Griffin, et at,, 1983 
Kean,, 1983 ,. . . ■': ' 
Dickey n.d.l!)) 




Pattofi.ra. fcn ttie evaluator 'ttes direct, 
personal responsibility for getting information. to 
tlie rijtit people," the utilization potential is 



,Alkin, 0aillak,S|ite,j5?9, taevaluator' 
philoscpliy that stresses' "the importance of having: ' 
the evaluation serve the project and not just meet ' 
the pro forin.roquirciiionts" helps engender trust. 

Ihts factor is associated with^siidience orientation , 
and Mr'pdispositiontoi evaluation!^ 

taedy,lffi. Beyond taktnglthe'stance that his or 
her job is "to serve," the [in house] evatatorm^y ^' 
not be able to define his/her role; "role is 
detertned to a large extent by the organizations 
theyW' (p. 161). Successful evaluators will , 
adopt a problem-solving strategy that is compatible 
with their distrtct's'problem^solving strategies. ^ 

Wctey, n.d.(a) To ^enhance utilization, "evaluators 
, should adopt a more collaborative rale, involving the 

decision-maker and the staff'in'decisions about the ' 
evaluation" Ip. 241. 



/ilkin,DaillaC!Nhite,\ro, [valuator rapport 
with his or her audiencel will; generally contribute 
totheuseof infoniiation(p.'2«). 



Guba 8 Stufflebeam, m 

'«olleyS tee,457r ' 
Connor,, 1579 . ' 
Bolley, 1579 
ffln Uaw. m ■ 
te'chSCordray,m 
Wey, M " 



Apling Stenedy, 1951 

Oraverman, 1981 •; 
Hansen, Nartin,S Word, 1981 
Leviton 8 Hughes,- 1581. . 
0'teilly,ffl 
Stevenson, 1981 
lipson, 1981 
«eiss,,1981 

,8 Pechman, 12 
in,et3l.,W ' 
;.19S3 ' 
Dicteyn,d,(b) ' 
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tasonS Sherwood, 19I2' 
fflin,i975 . 
AlkiniLaw, 
Cooley, 1980 



1. FM 

j > 


2, ffiSEM LITER/iTK 

c 


3. THEOKETICi/COfiCEPTU/lL ' 
LITERATURE^ 


UX ml 


■Braskamp i Brown, 1980,. To the extent that the 
evaluator demonstrates understanding of the^prograin, 
and Its internal and external political environient, 
rapport Is enhanced and the potential for utilization 
IS increased (p. 93), 


i ■ 
J V 

■ '' 


' 5. Understanding 
political consi-,, 
derations 

^ '0 

\ 


Braskamp S Brown, 1980. The'evaluator who , 
demonstrates political understanding is in a better ' 
position to work toward utilization (p. 93). 

Cichon i D^er, 1982. "Evaluation is'only one part 
of the onjoing social and political context of ariy 
educational program, and, as such, it can either ■ 
become a part of the system from the gr-ound up or it ' • 
can be left to chance to fit at the end". (p, 23), 
This factor is associated with evaluator role, compo- 
sition of audience and its organizational context, ■ 
and sources of information beyond the evaluation. 

King S Pechman, 1982. Successful collaboration. In 
the Interest of utilization, is difficult and need 
not Mance utilization. Political factors fiey 
preclude collaboration; users may view the need for : 
collaboration as an indication of poor evaluation 
management (p. 23). " - . 


1 

Jlkin, 1975 •• 
Cox, 1977 ■ 
Granville, 1977 
Brandl;1978 
•Brickell..l978 
'Pelz,19?8 
Campbell, 1979 

Holley, 1979.. •.--— 
ilkin S Law, 1980 \ ■ 
Cooley, 1980 

Hansen, Mn, & Oxford, 1981 
O'Reilly. 1981 
Weiss, 1981: . 
King, Thompson, S Pechman, 1982 
Kean, 1983 . ■ : 
Nagel ,-1983 . 

I^adzikowskl i Seaver, 1983 ^, 
Dickey n.d'.(b) ■ jv 


1 
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s ■ . ■ 
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6. Credibility 



I.Jackpndand, 
identity 
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Browfi & Braskamp, 1980. .Credibility is an' important 
detenninant of utilization. Credibility is related 
to tlie trust program staff places in .the evaluator, 
and to their perception of how i^ll he or she . 
understands their program ip; 93). 

^Alkin, Sfecher, S Geiger, 1952. These case studies 
clearly identified "evaluator reputation and credibi- 
lity" as a relevant factor in evaluation use (p, 
82). This factor^ is associated with evaluator role, , 
audience rapportjple'/ance of infonnation provided, 
and audience predisposition to evaluation in terms of 
fostering utilization, 

Reisner et al, 1882.' "Fdrcefulness and earned 
respect" seem to typify credible evaluators. This 
determinant of utilizatHm is associated with such 
personal , evaluator chariicteristics as his or her 
desire to serve the needs of th.e program being ' 
evaluated (p, 59), 



Braskamp, Brown, 8 tlewnan, 1978. While the title of 
the evaluatar can influence audience perception of 
the overall objectivity of the evaluator, title does 
not seem to affect audience 'agreement with the recom- 
mendations the evaluator makes (pp. M9).^ This ■ 
factor is associated with credibility, canposition of 
the audience, and their information needs. ' 

flew, Brown, & Littman, 1982. The sex of the 
evaluator mciy have variable, effect depending upon 

factors associated' with the evalution report and 
factors associated M the audience (p. 18). Sex 



lllkin,1973 
Windle I Bates, 19H 
Alkin, 1975 ^ 
Guba, 1975 
Cox, 197? 
l^olley, 1979 
Alkin & Law, 1980 
Stalford, 1980 

Hansen, Martin, S Oxford, 1981 
Joint Coiittee on Standards for 
Educational Evaluation, 1981 
LevitonS Hughes, 1981 ... 
O'Reilly, 1981 
Stevenson, 1981 • 
Thompson, 1981 ^ 
King, Thompson, S Pechman, 1982 
Griffin, etal, 
Oickeyn.d.lb) 



Alkin, 1975 
AlkinS Law, 
King, Thompson, & Pechman, 1982 
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2. , MCH LITERATURE 



h THEORETICALflCEPTUAL 
LITERATyRE ' 



bis Is likely to be subtle; I.e., wtien audiences "are 
judging unfamiliar material, or of an unfamiliar 
field, they are more likely to make stereotypical 
responses. 



Alkin,Daillak, | Wtiite,ro. There are often 
inultiple audiences for an evaluation, or parts of an 
evaluation, Itiis factor is associated with evaluator 
role and allegiance to selected audiences (p, 253), . ' 

Kennofc/1902. "...to the -extent At different 
partictpan1;s define issues differently, a given piece 
of ep'dence mciy be perceived as irare or less' relevant 
to different participants and as equally relevant but' 
differentially Instructive" (p. 33). • 
/ This factor is associated with evaluator role, 
/aiiiount and type of contact, sources of information 
' beyond evaluation, and audience interest in 
evaluation. 



Alkin, 1573 
Alkin,ro 
Cox, 1977 

*e,ra ■ . 

Honey,1979 
Alkin S Law, 1900 ^ 
Cooley,1980 
Bravemian, 1981 ■ 
.Hansen, Martin, & Oxford, 1981 
O'Reilly, 1981 
Stevenson, 1981 
Altschuld, etal, 
Chelimsky, 1983 
Kean, 1983 , ■ 



Peck « Triplett, 1983 
Radzikowski k Seaver, 1983 
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b, orpizatlona] 
. positions 



c, professional 
experience levels 
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Alkin,Kosekoff,Fit2gibbon,&Sel1p,1974, The 
evaluator's perception of the relevant decision . 
nl^ri - those in a,position to act upon the. infor- 
mation, can influence th^ usefulness of the informa- 
tion for decision making (p. 8S). Ihis factor is 
' associated with evaluator role selected, evaluation 
processes selected, the kind of information selected, 
and how it is reported, , 

King & Pechman, '31 Utilization "requires the ex- 
plicit and continual action-based support of powerful 
administrators" (p. 32), But this support by itself 
, does not guarantee use, and some people will use • 
evaluations even in the absence' of 'urging by tl^air 
superiors. . 

This factor is associated with political 'jnder- . 
standing, the requirements for the evaluation, orga- 
nizational considerations, and the evaluation ques- 
tions bf interest. 



Neiian, Brown, & Littman, 1982, Persons "closest to" 
the decisiqn-making process" [adninistrators] appear 
to be "more critical [of the evaluator] than rovices" 
(p. 18). This factor is associated with evaluator 
role, audience for the report, jnd composition of the 
audiences. . o ' 



Brickell.P 
Meltsner, ffi 
CookSPollari,19n 
Holley S Lee, 1978 
Resnick, et al , 1979 
Alkin & Law, 1980 
O'Reilly, 1981 

King, Thompson, S Pechman, 1982 
Altschi!ld,etal,1983 
Kean, 1983 ■ 



Alkin, 1973 
Holley S Lee, 1978 
Connor, 1979 
Alkin Uaw, 1980 
Radzikowski I Seaver, 1983 
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I Interest in the 

a. views about tlie pro- 
gram being evaluated 



b, program questions 
and concerns 
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' f idienro vle'^;; toward and 'Conltment 
to a progiui ^5 sui.i |j,at dec1s, )ns about the 
program have been made before tlie evaluation data 
have ten collected. Should the audience favor the 
program, and expect evaluation results to be posi- 
tive, then it will be dii;ficult to' build a utiliza- 
tion potential into evaluation information that njns 
counter to that position (p. 80). 
. This factor is associated with political under- 
standing, the role of non-evaluation information, the 
audience's perceived risk, audience openness to 
change, anli evaluator'conitment to use of results. 



Alkin, Daillak, S White, 1979. Potential users' of an 
evaluation m have iny varied (|uestions or concerns 
about the program being evaluated, This factor is 
associated with audience composition and their pre- 
ferred forms of information (p. 238-239). 



Braskamp Hrown^ 1980. Discussion and negotiati5|, 
between evaluator and program staff should-address- 
the purpose of the evaluation, and how its results 
might be utilized. The earlier the staff is involved 
in such deliberations the greater the .potential for 
utilization (p. 92). 



Mk , Mar:iii J Oxford, 
O'Rei'ily, 1981 



Rodiian Holodny, 1964 
Bauman, 1916 
Cox, 1977 
David, 1978 
HolleyS Lee,. 1978 
Holley, 1979 
AlkinUaw, 1980 
Cranbach et al , 1980 
Leviton'&Boruch, 1980 
Braverman, 1981 
Hansen, Martin, I Oxford, 1981 
Leviton & Hughes, 1981 
Stevenson, 1981 • , ' 
Studer; 1981 
Thompson, 1981 

King, Thompson, i Pechran, 1982 




Wllak, 11, Sctois often need Momctton Ofi a 
set of issues ttiat ^ lie qutte different froii, and 
less aiitious than, ^at tke evalyator ndjlit ' 
expect. Tliis factor is associated M audience 
expectations for tlie evaluation in tens of Uie 
potential for utilization Ipp. 153-155). 



answer questions raised from inside tlie . program 
.uNlien the questions jutdinj a sWj come'from tlie ; 
program personnel themselves, there is an interest in 
'and coiitment to the inquiry engendered ky the . ' 
ownership,"' (p. 241, Ihis factor is associated wiih ■ 
e\'aluator view of user involvement, amount and ijpe 
of contact, and reporting format and substance. . 

tanedy, 1582, If a variety of potential users are ■ 
•involved, use is litely to increase when they can ' 
"agree to modify tir positions on a troad class of 
events and practices, so that each could change a 
variety of decisions and actions in the future... 
Ilgreement [is Wy to be let] by modifying the . 
message presumed to, have come from the evidence" (p. 



This factor is -associated wi'th evaluator role, 
user involvement, and coiuiiication of infonnation,. 



,etal, 1 



PecUjriplett, 1383 
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I FACTOR 



I mkm LITERATURE, 



c. urgency of questions 
and concerns 



i predisposition to 
/evaluation/the 
' evaluator . 



ing forthcoming 
generating data 



Alkin, Daniak,;&11te, 1979. The urgeiic/ associ- 
ated with an evaluation may range from assuring that 
a report is subfitted by a certain date,,to address- 
decisions by program authorities, to 
to help resolve staff conflict (p. 
238); To the extent that an evaluation is to address 
multiple questions, and to the extent that the 
urgency of the iuestions varies, the evaluator imst 
determirie wltich are the, most pressing issues. If the 
-issues are not iealt with, it is unlikely that the ■ . 
evaluation will be used to answer any questions. 



Anderson, Ciarlo, Urodie, 1981. "...the strength 
and nature of preexisting feelings about an issue, msiy 
be expected to strongly influence the potential 
utilize 
conside 
offers 



s Interest In working with the evaluator and 
ing the infomiatlof! that t j ^^^valuator 
(p. l22). . . V.. ' : : 



(King S toman, 1982. To enhance utilization poten- 
tial, tie evaluator will need to gauge^the existence 
and degree of disharmony behind users' "espoused - 
theorist [stated beliefs] and "theories in use" 
[actioni] (p. 27). i 

This factor Is associated with political 'under- 
standir|g, information dialogue, and user personal 
characjeristics.' • 



3. THEORETICAL/COfJCEPTUAL 
LITERATURE 



Holley !i Lee, 1978 
[loney, 1979, -, 
Hansen, Martin, & Oxford,. 1981 
0'R^lly,19pr 



Davis i Salasin, 1975- 
Forgione, Kaplan, S Orland, 1979 
W!ey, 1979 ■ 
Alk1n& Law, 1980^ 
Coolcy,1900 

Kennedy, Apling, yeuman, 1980 
Crowley, 1981 

Hansen, Martin, I Oxford, 1981 
Levitoni Hughes, 1981 
O'Reilly, 1981 , ■ 
Stevenson, 1981 

Thompson, '1981 \ 
King, Thompson, & Pechman, 1982 
Nagel, 1983 ' 
Dickey n.d,(b) 



1. FOR 



2. raCH LITERATURE 



. 3, THEORETICAL/mCEPTUAL 
LITERATURE 



e. perceived need for 
the evaluation 



f. expectations for 
the evaluation 
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Brown, tan, I Rivers, 1980,. Persons with a high 
perceived need for evaluation information tend to be 
more satisfied M the evaluation and thus the 
potential for evaluation is increased. This factor 
is associated with evaluator role and credibility 
(pp. 67-73). 



Alkin, Datllak, & White, 1979. The evaluator who is 
not tuned 'to program expectations is likely to lose 
that audience and the potential for utilization is . 
diminished. This factor is (issociated with' evaluator 
role, rapport, and evaluation processes selected (p. 
238). ^ 

King & Pechman, 4982. The evaluator cotimitted to use 
must understand that use msiy be "signalling" (infor- 
mation used to report optside the system) or 



Cox, 1977 
Rich, 1977 
Dobson, etal,1978 
Holley Uee, 1978 
P'^lz, 1978 
h io, etal,I979 
Holley, 1979 , 
Young S Canptois,^ 1979 
AlkinUaw, 1980 . . ^ 
Hansen, Martin, & Oxford, 1981 
O'Reilly,. 1981 • : 
Thompson, 1981 

King, Thompson, i PechfiSii, IM 



Cipbell,1969 
Carter, 1971 
Kiesler, 1971 
Alkin, 1975 
Davis fSalasin, 1975 
Rein Hhite,. 1975 ' 
Granville, 1977 
Brand!, 1978 
Wz, 1978 
Studer; 1978 
Holley, 1979 ' ' 
■Rich, 1979 ' 
Alkin Uaw, 1980 
Cronbach, etal., 1980 
Leviton & Hughes, 1981 
Weilly,1981. 
Stevenson, 1981 
Studer,1981 , : 

Weiss,'1981 / : '^^^ 
King, Thompson, & Pechman, 1982 
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1. FACTOR 




3. IHEOr.£TIC/\L/CONCEPML 



I.B.2,f. cont. 



g. risks of tfie 
evaluation 



3. CopMot to use of 
evaluation results 
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"charged" (Infomiatlon used Inside the system (p. 
36). While signalling often Involves no iicre than , 
tfie perfunctory submission of a required report wliich 
has little internal relevance, charged use suggests'/ 
internal action it information use. Charged use may 
be "instrumental" - an action directMnked to a ' 
specific piece of information; "persufcve" infor- 
mation used for personal ends;, or "conceptual" - 
information leading to a pdual change in someone's . 
thinking over time, [valuator understanding of ,the 
audience's intended uses can enhance utilization."' 



Alkin, Daillak, Ohite, ,1979. , There are indijlldual 
and institutional' risks and potential .benefits! in 
conducting an evaluation. Users, depending onj their 
orientation to evaluation and their expectatidns, 
will make decisions about costs and rewards which 
influence the extent to which evaluation •information 
is used and the uses to ^ihich it is put jp. 250).' 



Alkin, Stecher, & Geiger, 1982. The attitudes of 
dec-i'sion makers, toward using evaluation is a factor 
strongly related to evaluation use (p. 85]. This 
factor is associated with audience predisposition to 
evaluation and their openness to its use. ! 

King I Pechman, 1982. The evaluator ^tio acts |as if 
his wk will be used by a "rational decision 'maker" 
who will make "big bang! decisions should not ^expect 



Chelimsky, 1983 
Griffin, etal, 1983 
Kcan, 1903 • 
togol,1983 



Connor, 1979 

Holley, 1979 : 
Alkin Haw, 1980 i 
Hansen, Martin, h Oxford, 1981 
O'Reilly, 1981 .\ 
Thompson, 1981 \ 
fJagel, 1983 ^ \ 
Dickey n.d.lb) 



Brickell, 1974 ■ 
Fairweather, etal, 1974 ! « 
Alkin, 1975 

Caplafi, t?t^il, 1975 I 
Glaser, 1976 
Chelimsky, 1977 . 
Cook, 1978 
Lynn , 1978 

PfefferSSalancik,1978 
Alkin, Daillak, S White, 1979 

Hall, 1980 

Hansen, Wn, S Oxford, 1981 
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1. ■ Professlofial styles 

a. adninistrative afid^ 
orjafilzational 
styles ,^ 
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to l)e disappointed if tlits kind of utilization to 
not occur (pp. 1M8). is factor is associated 
with evaluator role, view of user invotant, pro- 
cesses selected, and coiunication of infonnation. 

Msner, etal., 1552, 1 significant decision 
mater's support of evaluation . , - tads to Increase 
utilization Ip. 60). 

Dickey, n.dla). ".ntlie project director Is nDst ' 
likely to use the evaluation *n lie or' stie is - 
interestedifiandcoiittedto-it..."'(p. 



lin,Ba111ak,8ite,lW5. Utilization of evalu- 
ation Infonnation Is enhanced ilien tfie tcision rter 
tes the ainistratlve and orgafiizational skills 
necessary to "jet things done," to start action 
[and] to carry it through to successful conclusion 
(p, 255!. ' 

Cichon 8 Dwyer, 1982; "n.manageient and evaluation 
functions can and should lie integrated in the prograi 
'leadership rale" (p. 23), To,that extent that 
educational managers incorporate evaluation findings 
in liieir Wly routine.activitles, the potential for 
■'utilization is enhanced. 



Ilmsky, 1383 
5riffin,etaV 



Cooley.lO 
Stalfort, 11 
Hollej S lotusel!', 
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2. raRCH LITERAIURE 



3. lORETICAL/CONCEPmi 
LITERATURE 



Kennedy, 1982, Information, rather tlian beifig the 
■principal contributor to a telslon, "Is Instead a . 
tool used well or readily by all participants, thus 
whatever [telslon] develops will not be deteniiified 
by the substance of the evidence but by tlie 
distribution of political and mipulative skills 
among participants" '(p. 67). .„ ' 
,. This factor Is associated with political under- . 
standing, the role of other infonnation) and the 
users' organizational positions. 



King & Pechman, 1982. "...self-confidence.Mand^ 
interest In and openness to evaluation [can] separate 
likely users from non-users" (p., 30). But 'this is 
n6t exclusively associaiedwth program advocacy; 
rather it suggests a personality who can take the 
initiative to apply theo'nformation at hand and who 
is supported by "a favorable political context and 
climate" (p. 32). ^ 

This factor is associated with user's risk ' 
perception, view of the program, evaluation 
expectations, and organizational factors, 



y, etal, 1983. "Principals i^o take more of 
a lead in directing the school seem to need and rely 
iiiore on infonnation about how the school or its 
programs are functioning than'those who are less . 
explicit ebout the qoals'and standards of the school 
lp.6)."~ , 



' / 

/ 
/ 

/ 



' / 

Holley,«' / . 
Alkin S Law, 1980 '/ 
Hansen, Irtin, S:Oxford, 1981 
0!Re111y, 1981 ' 
Kean, 1983 ' ' 
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2. KMLlTEmi 


3, IHEOREIIML/mEPm 

Liraw 


c. openness to new 
ideas or cfianje 

f 

r 


Alkin, Daillak, UMte, m Utilization is 
enhanced if the decision rter is open to new ideas, 
is able "to. mm probable benefits and risis, to 
weigh tbem against each other," and then to act 
accordingly Ip. 255). 

McColskey, etal, 1983. "tee is not much an evak- 
ator can do aboyt an aduinistrator's'laCk of open- 
mindedness, bpt role, perceptions and understanding of 
social science research methods are characteristics 
of aduinistrators that ^valuators can help to modify 
o.By helping'adniinistrators identify goals, means to 
achieve these goals, and iDsasures to assess progress 
toward these' goals, evaloators m^y encourage adminis- 
trators to act as'leaders.Mand to become better 
infoniiation users lpp.6-?)." ■ . ■ 

■ J . 


Hanson, Martin, J Oxford, 1581 ■ 
O'Reilly, 1981 
Studer, 1981 
fan, 1983 

1 

1 

1 

I 


5, Infonatlon processing 

1 

r 

imc 


n 

Alk1n,DaillaM'fttte, 1375. Users' views about . 
.acceptable forms of infonnation can influence evalu- 
ation use. Ihe evaluator *o feels tliat a particular 
question is best answered by ([uantitatlve data, and", 
who is working .with an" aintstrator io is inre com- 
• fortable working with (lualitative data, can attempt ^ 
to provide both kinds of infonnation or can attempt 
to convince the' administrator ;to accept 'and act upon 
purely (jiantitatlve data. . Ihe major point, however, 
Is that the evaluator find out *at the, acceptable 
forms^of information are (p, 239),, 


ta,197? 

Hansen, Martin, 8 Oxford, ffll 
O'Reilly.m 

loiing S Cooiptois,- 1979 . 
Spniiil] J Mrow, 1981 
«eiss, 1978 
itzber9,1973 
lla§el, 1973 ■ • 
Holley,1983 , 

1 




II, CONTEXI FACTORS 
K Preexisting Evaloation Bounds 

1, Written requirements ^ 
for tfie evaluation 



2, Other contractual 
■ obligations for tlie 
evaluator . 



3, Fiscal constraints 
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Alkin, Dalllak, S Wfiite, 1979. The evaluator and 
program managers niist discuss the range ^of require- 
ments - specified data, analytic requirements, re- 
porting needs --.that' emanate inside and outside the 
project, To the extent that there are conflicts be- ^ 
tween th.ese requirements, or that one, set of^equire-^. 
ments receives preeminence, the likelihood for 'utili- 
zation may be reduced (p,237V 



Alkln,Daillak, SWte, 1979. In addition to legal 
requirements, program managers themselves "may estab- 
lish " In virltten form or. otherwise - other tilings 
that an evaluator must do" (p. 237). ■ These obliga- 
tions iray go far beyond state/federal requirements. 



Alkin, Oaillak, & White, 1979. To an extenti fiscal 
requirements can influence the role of the evalua- 
tor. This'factor Is associated with evaluator poli- 
tical understanding and the substance of the evalua- 
tion information (p. 237). " . 



Alkin,Uaw,M • 
Joint "Comnittee on Standards for 
Educational Evaluation 1981 
Weiss, 1981 



Alkln, l973 ■ 
Alkin Raw. 1980 ■ 
O'Reilly, 1981^ ' 



Holley,,1979 

Alkin ilaw, 1980 ^ / ■ .. ^ 
Hansen, Martin, & Oxford, 1981^ 
Oickeyn.d.lb) • 
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■6. Orpizational 
Ctiaracteri sties 



1. Ifitraorjanizational ; 
features ^ 



a, role of central/ 
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b.'InterrelatiofisMpi 
tetween ufiit and ; 
central/district, 
administration •' 
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Alkin, 15?3 ' 
lin,,1975 

Davis l/^asin, 1975' 
Meltsner, 1976 

Co)f,1977 ■ ' 

Glaser.OT 
rtr,etal,1977 
liolley Ree, 1978 " 
tofinor,'1979 . 
Stalford, 1980 

Hansen, Martin, S O)(ford, 1981 
Stevenson, 1981 
Studer,198r : 
tapson, 1981 ■ 
Altsctald,etal,'1983 - 
Griffin,,etai, 1983 • 
Dictey ni(bl 



Alkin,'Dail!aUWfiite,197"9. If tlie organization 



•choice of role/evaluation-^fOGl.-the-evaloator^is™ 
able to serve project needs, thereby enhancing tlie 



/ilkin, 0a111ak, & White, 1979, District responsive- 
ness to„ the local projp and its needs can influence 
the extent to ^Mch the evaluator and program admini- 
strator can cooperate*to conduct a locally useful 

evalu^tion(p.2C 



Boruch S Cordray, 1980 
0'Reiny,198i 



Holley Kee, 1978 " ' 10^ 
O'Mlly, 1981 J"*^^ 




c. 



arrangements 



d. unit level aiitonomy 



Cr- sources of Infomia- 
• tron beyond evalua- 
tion,: likely to be 
in use ■ . 
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3. ■ THEORETICAL/CONCEPTOAL 
. LITERATURE • ' 



Alh'ii, Dalllak, & White, 1979. District institu- ' 
tional arrangements can affect the conduct of evalua- 
tions, For example, in a situation viiere a central 
evaluation office ser'^'es all programs, an "internal ' 
evaluator l^v^y have to convince tlie local site of ttie 
usefulness of doing an evaluation. On-^the other , 
hand, an "external" evaluator hired for a specific 
evaluation task my have' to convince the central 
office and/or local site of the relevance of his work 
!p..248l. 



Alkin; Daillak, & White, 1979. Autonomy, within and 
among schools can provide the evaluator and tJie 
program developer "the ffeedo| of action necessary to 
decide,., hoW' best to use the evaluation" (p. 248). 

HcColskey, et al, 1983. "Principals vjho agreed [that 
they can make] important decisions in the school and 
[have] the- power to affect changes in the school 
reported that they relied more on.., information than 
those who [do not have such freedom] (p.6)." 



Patton, et al., 1975... Evaluators who understand that 
their "results [provide only] another piece of a ' 
complicated infonnation puzzle," 'and who apply that 
understanding, are likely to' produce infonnation that 
has potential for utilization. 

Alkin, Daillak, S White, 1979. .Site-decision makers 
alwc(ys'hav6 access to information beyond tliat .' 

supplied by the evaluator. The extent to which other 
information sources are perceived as mseting the 



Alkin Uaw, 1980. 
Cooley,il980 

Hansen, Martin, ^Oxford, 1981 
O'Reilly, 1981 . ' 



Alkin Uaw, 1981 
O'ReillyjM- 



^Caplan, et al, 1975' 
Rein S White. 1975 
Chelimsky, 1977 ^ 
Cox, 1977 

Hplley Uee. 1978 
Boruch &'Wortman, '1979 
Connor, 1979 
R1ch,;979 . >. 

Alkin & Law, 1980 
Hamilton, 1980 , 
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2, RESEARCH LITERATORE , 


I THEORETItAL/COCEPM 
_;LITERATURE 


» 

II.B,l.e» cont. . v • ^ 

« 


.1.1-.-^ ■.■^....-^i— I. ...... , ,.. ,„, ,. „, 

users perceived^inforpation needs has impact on the 
ultimate acceptance and utilization of evaluation 
(p. 249), Evaluator role and 'desire to involve users 
^in th^ evaluation are associcitedjth this factor. . 

• 

1 

• 

ft ' ■ 


Braverman, 1981 

ten, Wn,X Oxford, i981 ■ 
■'Levitonfiflughes, 1981 . 
o;Reil.ly, 1981 
SproulHZubrow,"198i . 
Weiss, 1981 

Ktn^,'Ihompson,|Pechman, 1982 
Holley, 1983 ; • . 
NageU983 ' 


' . f. perceived Ifistitu- 
, , ' tiofial risk . 

/ 

• ■^ ' . 

t 


» 

A]kin,Dainak,lWhite, 1979. If Uie organizational 
sense is that^ tiiere are greater risks associated wjth 
th& evaluation •than-liiere are possible benefits,' ' ' 
^ there is less likelihood of the evaliiatioa, being ' 
' used. . - . , . 

o' 


\ ■ ' • 

Holley, 1979 ' 
Alklfi Uaw, 1980 ' ■ ■ ' ' 
O'Reilly, j981 
Thompson, 1981 

' 1' "f 


■1 

2. Extraorganizational '^-^ 
features 


1 


, 1/, ■ ' . 

■ 1 ' • ■ ^ 

1 * 

/ ■ 
M 


a. comnity climate 

> 

1 ■; 

f . 1 

; 

4 


Alkin, Daillak, « White, 1979. School-coiunity ' * 
relations can range from neutrality to highly 
politicized, partisan settings, Jhis factor,' which 
is associatet! with evaluatOf understanding of 
political considerations, can affect receptivity for 
utilization, and bears on evaluator role, user^ 
concerns, land evaluation procedures and reporting 
metho,ds (p. 236). ' 


Cox,\97L^- 
flolley, 1979 . 
Alkin Uaw, 1980 
Cooley, 1980 

Hansen, Martin,^50xfoi\f, 1981 ■ 
O'Reilly, ' 
Thompson, 1981 • - . 
King,'Thompson,1p£Cten, 1982; 


b. coiufiity influence 

■ ERIC ' 


Jlkin, Daillak, ,& White,. 1979., Where.the local com- 
munity has a stake in the program being evaluated, a 
necessary step In- creating the environment conducive 


Alkin, 1975 ■ 
Cox, 1977' 
»y, 1979 
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c. role of other 
agencies 
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2. MeHLIMIURE 



to evaluation receptivity i^to enlist comiunity sup- 
port Ip. 218). 'This factor is associate(i\wlt!i IJie 
cyoluator's role, credibility, and plltical under- 
standing. Since, the a(iTiinistrator will be important 
in enlisting cownity suport, his/her a(iiiin1strat1ve 
,and organizational skills, as well as initiative, ' 
als'o bear on this factor, ' 

Reisner, et al., 1982. "Utilization increases. „when 
the local cownity expects to be involvEd" (p, 60), 
Ihis factor is associated .with evaluator philosophy 
about user involvement, .evaluation conijient to 
utilization, and local decision Ekers' attitude , 
toward evaluation, 



Alkin, DailTak, S White, 1979. ' Above and beyond ini- 
tial .mandated requirements, other government agencies 
can continue to influence the conduct or focus of an 
evaluation after Jt has begun and can hava an impact 
on the eventual utilization.of evaluation information 
(p. 251), this factor is associated with the role of 
^the evaluator, the composition of his audiences, :the 
evaluation processes selected, and the substance of 
the valuation report. Utilization may dimimsh to 
the extent that the evaluator maintains a static 
position in a, situation the influence of other 
government agencies is c|ynamic. ■ 



I JEOREIICAL/COfJCEPIOAL' 
:■ ■ LITERATURE 



Alkin 8 Law, 1980 
Copley, 1980' 
Stevenson, 1581 



Holley, 1979 
O'Reilly. 1981 . 
Stevenson. 1981 
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. C. Program Cfiaracteristics 
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1. Age/maturity : ' 
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ll(1n,D5illaU Wte,ro. These case s,tu(lies 
suggest tliat newprograis useevaloation information 
for public relations purposes and for building 
external support for tiie program ..(pp. M; 
217-220). This use is associated witti community 
influence, composition of the audience, program' • 
questions and concerns, and administrator styles. ' 

New programs"'also' find evaluation useful in ini- 
tial curriculum, :^staffing, procedural' and other 
" "infonnal" decisions because there are fewer-vested 
intere^sts or>teacher and ciiiiiinistrator resistanceoto 
overcome (pp,'67-]3; W5-147; 206). Associated' " ■ 
factors here are evaluator personal characteristics, - 
Information 'dialogue, amount/type of contact, sub- ; 
, stance of evaluation information, aWnistrator ! 
style, and program/ (]uestions and. concerns. 


■AlHn,.l973 ' 
Holley, 1979 ; ' 
Weiss, 1981 

Griffin, etal; 1983, . \ 

.<v> f - . ■ / 


/ 

2, Jnnovativeness ,. , 

/ ■ ■ ,, ; 

1 ■ 

0 

> • 
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'■ All(in, Daillal(/fi White, 1979. Programs W are ■ 
unfamiliar to staff and coiunity, and that entail 
radical ch'anges.in .previous practicesj draw on evalu- 
ation inforniatjiop to clarify initial confusion about V 
content,, direction, procedures, and responsibilities 
Jppi;' 5U2, W5)v This utilization is associated 
^With: evaluator (person?! characteristics, amount/type 
"Of contact ?<ith users; substance 4 evaluatiofi 
■ information, and program questions and coiijE^iSt-L^^ 

Where the innovation is 'schqo>wtdfrrt 
information ^ be. considered superfluous until 
things,"settle down" of their own .accord (pp, M, 
. 212). "Ihis disinclinatiori to litifization appears 
■mos1;jtrongly associated with administl^ator style and ' 
his/her questions.and concerns' about the progrd 
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c ■ ■ 

201 

' ■ ■ ' ■ '' ' 

■ ;V • ^. ^ ,. r , , , ■ 

[ \ " *. ■ ■ ■ ■ ' ' ' • ■ 



• I FACTOR 




3. mETICAL/COHCEPIUHL 
■ ■ LITERATURE ' 



••^ Overlap with, other 
' ■ 'programs , • 



III.E' 



A. Informatlofi Dlaloye 



' > 0 



■ : • 0 



%^ Aiiount of interaction 
between evaluator and 

,, users 
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Alkin/Dainak, '& White, 1979.' Utilization iiy be : 
related to program overlap, particularly if pne pro- 
grai is a, component of a larger pgram. Related 
factors are iiethodological rigorysubstance of evalu- 
'ation information,^ relevance to' user cpncerns, fiscal 
constraints, site-level autononiy, program (ijestlpns ' 
and concerns, and the adminlstratorVorganizational 
skil'ls. • , 



Alkin, Stecher, Keiger, 1982. Utilization is , 
. increased* if the' evaluator makes serious attempts to 
" involve his /audience . in siich tangible evaluation pro- 
' ducts as "how to read, analyze, and m decisio'nsv 
upon evaluativfc data" !p. 91); for example, the data 
in' a report. This factor' is associated, with evalua- 
tor role,'' his view of audience involvement and-com- 
mitraent/tp utilization, and his understanding of 
audience lieeds within the particular organizational 
setting. 



Braskamp 



i Brown, 1980. The earlier the evaluator 



possible 



and program staff 'meet to address the subject of 



utilization the greater, the potential, for; 



utilization (p. 92). 
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B. Sutece of Evalyatiofi 
' Infontion 

1. Infoiiatiofi relevance 
forosers : 



^ Wersoo, Cfarlo,'.S Brodle, 11. "v..ta,rachone \ 
' can. affect Je degree of caring or concern about an 
issue iy very well te tlie prtnary .deteiinant of , ■• 
future changes in [utilization] behavior or practice" 
(p. 122). This factor is associated with audience 
expectations for and predispositions to evaluation, 



reisner,etal., Mr "Evaluation use increases 
when decision iters are assisted in understanding . 
how they might use evaluation data; (p.. S5l. ■ 
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,DaiHak, »hite,m Jhe rare the focus of 
the evaluation findtng/recoiendation reflects an , 
important program concern, the irore it is likely to^ - 
beusedln dectsioniking lp. il. This factor is 



a^socia^ed with evaluator role' and the identification ^ Uielson, 19/5 



of userl concerns. 




■ "...there is rot necessarily a 

direct relationship between improved writien reports 
and improved levels of use,; If auser finds ritten ' 
infokiation of value to his particular situation, he 
Is Mely to use it, regardless of 1t^ 
fof! wljatever reason, ities not see^^ 
valuel^ he will not uselV!# 

Ws factorjs associated with political and ' 
organlzatlonal'factors, and'rtience composition. , 
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Dickey^ ni(a) "..:the project director is most 
likely to yse tfieeva]!j.i!tioiita.nits 
recordations [are] helpfu],,." (p, 17), 



Braskamp & Brown, 1980. ImportiTprpOecisions 
are often based on a broad range of information 
needs, To the extent that specific evaluation 
information addresses these heeds, the|gher''tfe 
potential for utilization (p. j3)..--" 

Kenne(ly,-1982r"Mrif'« that ■ 
-Infomiation can have nultiple meanings, then one ^ ' 
introduces' a host of questions about the difference 
between its factual leaning and its normative 
message, and about its potential to make a . 
substantive contribution to the forthcoming decision!' 
(p. 66). : ' . \ 

, (Kennedy! 19811.1 This factor is associated with 
evaluator understanding of political considerations, 
relevance of information, and ciposition of" the 
audience. 
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Kennedy, Splinj, S iewn, 11. iile the effect of 
timing of a final, formal evaloation report on 
utilization is difficult to gauge, timely and well 
focused formative reports can increase tlie potential 
for utilization. 

ticton SD^er.'lK. in using 

tlie results of an evaluation begins wiien the results 
come in, you're probably too late. Use Issues must 
be addressed from the start of an evaluation to ixt- 
mize tlie potential" (p. 231. This factor Js associ- 
ated'wi'th evaluator, rale, h of desirability of 
user involvement, and user expectations for and pre- 
disposition td. evaluation. . 



Alkin, tecoff, Fitzgibbon.'SSeltpan, IW, Eval- 
uations which provide timely formative informattcfi - 
e.g., which help project directors identify possible 
problems and *tch provide recoiendations for pro- 
gram changes - Increase the potential for utiliza- 
tion Ip. 871. This factor' is associated wi'th'the com- 
position of the audience and understanding of poll- , 

tical considerations in terms of likelihood of 
utilization.. 

David, 1578. "The timing of an.evaluation can by 
itself restrict the potential utility by rot meshing 
with the timing of the program planning" (pi 38). 
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Dickey, n.d,(a) " the project director is irost 
likely to use the evaluationo.when its final report 
is produced on time for the project's needs" (p. 
171. 



Braskamp S Brown, 1980. Using appropriate means of 
informal presentation can enhance utilization. This 
factor is associated lyith evaluator rolej audience's 
orientation and irtvolvement, the evaluation proceses 
selected, and evaluator coiiti^ent to utilization' 

Alkin,Stecher,.iGeige\l982..U^^^ 
creased to the extent thevevaluator selects presenta- 
tion methods and formats Ajch are appropriate to his 
audience and which clearly display answers to audi'- 
ence questions and needs (p. 89). 



Browni Braskamp, i Newnan, 1978. The amount of' data 
and jarp in an evaluation report can affect the 
audience perception of its technicality and diffi-- ■ 
culty lp. 339). ^-^his factor interacts v/ith evaluator 
credibility, audience composition/orientation, and 

preferred modes of information in the potential for ; 
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Fonnat of reports 



Brown, Braskamp, S Newman, 1978.. Jargon-loaded . - 
reports are likely to be seen as'irore "technical" and 
"difficult" than jargon-free reports. However, the 
effect of jargoh iw be mitigated by the inclusion of 
data v^ich help the audience to interpret and under- 
stand the information {p. 339). This factor is as- 
sociated with evaluator credibility, audience compo- 
sition, and preferred forms of infomiation. 



Reisner, et al., 1982. Graphic, narrative, and non- 
technical npdes of presentation, especially when they 
describe program strengths and weaknesses and give 
recoimenclations for program change, increase the uti- 
lity of evaluation information to local decisioa 
makers (p. 65). ■ 
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■Alkin, Daillak, fi White, 1979. Selection of proces- 
ses that are appropriate to. program context and audi- 
en'ce:orientation will help enhance the potential for ' 
utilization (p. 240),' ' . 

Dickey, n.d.(a) . "...the project director is nost » 
likely to use the evaluation when he or she.. .sees 
•its.procedures as appropriate" (p. 17)., , 
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Holley Uee, 1978 

Caplan^ 1980 •■ .'' " 
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Cichon S0w}er,.1982, "unquestioning adherence to ' 
[strict research] procedures ^ well jet in the wc^y 
of usefulness for program improvement" (p. 25), 'To ; ■ 
the extent that design, measureiiient, and analysis- 
procedures are sensitive to the purpose and scope of 
the evaluation need, the potential for utilization is" 
. enhanced. • ^ : . . 

e ' ' „ ■. ■ ■ ^ 
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m\\JtmjM%n ffHieevaluatioii' 
,stresses, or is pekeivei to stress,, mandated ^ 
Vequifeets, local perception of the evaluation's 
usefulness decreases as i^ll as the potential for 
utilization (p, 76). ^ 

Reisner et al, 1982. "Utilization increases to • 
evaluations are specifically designed (or are 
^ peri!!|tted.the i^exibility) to m local needs''lpr 
. 61). NJiiis factor is associated with the matter of ^ 
evaluation^ boundsj program organizational 'factors, 

and program staff Interest in evaluation. 

\\ ... ' 

Dickey, n.d;(a). ^ In the interest of enhancing , 
utilization,\evaluators should contract t«) undertake 
formative evaluations or suimiative evaluations, but 
'not both simult^ously" (p. 25),, . . ' 

Alkin, Daillak, & White, 1979.' In evaluator need not 
apply a formal evaluation m^del:,to enhance utiliza- 
tion., Rather.-utilization is; likely to be enhanced 

: to the extent that the evaluator Selects methods so 
as to 'be sensitive to the social/organizational 

■ context he is working in, and to provide a balance 
be'twecn treatment of mandated rei^uirements and the 
local audience orientation (p; 240). \The rolet 
selected by the evaluator wiH have a bearing on ;the 
procedures selected and their degree; of success. 

Lawton, et al, 1983. "If (b$s appear that no single 
or small set of strategies stands out as a predictor 
of evaluation success. Rather\ it «'M 
•-evaluators1use]Twldfwriety of strategies to , 

conduct an evaluation" ip.3).- 
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"^Ofganizlrig For Evaluation Use 





Evaluation Topic: 



Factor Cluster A: 



Pre-existing evaluation bounds 



User identity 



Program characteristics 



Intra-organizational features 



External features 



"WiflT res^p^cnoTny^^ keep^the Jollowihg in mind: 

Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



A-1. Pre-existing 
evaluation bounds 

^written requirements; 
other contractual obliga- 
tions; fiscal constraints) 



1. 




1. 


2. 




... ; 2. 


3. 


• 


3. 


4. 




4. 


5. 




5. 



.... -.."^^ 



A*2. User identity 
(range; organizational 
positions; professional 
experience levels) 



1.^ 

2. 

-3. 

'/.Vs.- 



1. 

2. 
3. 
4. 

5- 



\ 



\ 



Organizing For Evaluation Use 





i (Continued) 



Factor Affecting Use: 



Relevant Information: 



Things I C^n Do To Influence. Use: 



A-3. Program 


• 1. 


1. 


• characteristics 






^^(age; maturity; innoya- 






tiveness; overlap with 


2. 


2. 


other programs) 








3. 


3. 


* 


4. 


4. 




5. 


5. 



A-4. intraorganizationa! 
features 

(central/district office 

roles & relationships; 
I institutional an-angements 

& autonomy; other likely 
, kinds of information; 
^ perceived institutional 

& financial risk) 



1. 

2. 
3. 
4. 
5. 



o 

o 



1. 

2. 
3. 
4. 



A-5. External features 

(community climate & 
influence; role of other 
agencies) 



1. 




1. 


2. 


V 


2. 


3. 




3. 


4. 




4. 


5. ' 




5. 



.1^ 
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Evaluation Topic: *y\ 



Factor Cluster B: 



User interest in evaluation 



Oser commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



\ With respect to my own program and 'the above topici I ne^d to!;keep the folJovying,lnjTindi 



Factor Aff|epting°Use: 

Usdr interest in \ 
evaluation o 

(views; alSout program; 
questions & concerns; - 
predispositfbn to eval./ 
evaluator, need for eval.; 
expectations and risks" 
for the-oval.) 



0 * 



Retevant^lnfoPmation: 

o 

2. ; ,. ■ . • 

0 V 



Things I Can Do To influence Use: 
1. 

2. ' 

3. ( . 
■4. 

5. 



B-2. User commitment' 
to use 



1. 
2. 



Organizing For Evaluation Usp ' . 

■-.WotehSet B . (Continued) 



Factor Affecting Use: 


Relevant information: 


Things i Can Do To Influence Use: 


B-37EvaluatbrcHltr- 
acteristics (selected) 
(background/identity; 
credibility; choice of role; 
wtlMngness to involve 
user; commitment to use; 
political sensitivity) 


2. 
3. 
4. 


V • / 

, 2. e ■ 

■ 3, ./'^ 

. ../ ^ ^ - 




5. 




B^. Evaluation 


1. 




procedures— plan 


2. 


■ 2." ■ 




3. - 


3 / . ' 




4. 


4. 




5- . , ■ 


5. ~ 


B*5. User professional 
8tyl6(s) 

(administrative & organi* 
zationai skills; initiative; 
openness to new ideas) 


1. 

■. ■ 2: ^ 
3. 


. •■ ■ ,1. « . 

■ ■ ' ' ' ' • ■ . 

2. _ , 

3. ■ 



Organizing For Evaluation Ust, 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluatorcommitment to use ^ 



Information dialogue— formative 



User infornnation processing preferences 



C-1 Evaluation 1 
procedures — execution * 
(methods to be used; 
dealing with mandated 
tasks) 

3 



2. 



^47 
5. 



With respect to my own program and the above topicl 1 need to keep the following . in mind 
Factor Affecting Use: Relevant Information: \ Things I Can Do To Influence Use 

1. ■ . ' 

s ■ 

2. . 
3. 



5. 



C-2. Substance of 
evaluation information 

(specific information 
- selected; information 
relevance to audience) 



1. 




2. 


' 2. 


3. 


\ ' 3. 






4. 


■ 'i 


5. 


K 5. 



Organizing For Evaluation Use 

4u 





Factor Affecting Use: 



C-3. Evaluator 
commitment to use 



(Continued) / 

Relevant Information: Things I Can Do To Influence Use 



1. 

2. 
3. 
4. 



1. 
2. 
3. 
4. 
5. 



C-4. Information 
dialogue— formative 

=--=(amoant-&-quaiityof inter- 
action between evaluator 
* and user(s))' 



1. 


1- 


2. 


2. 


3- 


3. 


4. 


4. 


5. ' 


5. 



C-5. User information 
processing preferences 



1. i 




1. 


2. 


0 


2. 


3. 


\ 


3. 


4. 




4. 


5. 


V 


■ 5, 



Organizing For Evaluation Use 





Evaluation Topic: 



Factor Cluster D 



Evaluation reporting / 
Evaluator characteristj^s (selected) 



Information dialogu;^— summative 



User commitmenyto use 



With respect to myoyNn program and the above topic, I ne^fed to keep the following in mind: 



Factor Affecting/Use: Reievaf)t Information 



1. 



D-t Evaluation 
reporting 

(frequency;(timing; format / 
of report; statistical/nar- / *- 
rative data; format of / 
presentation) 



3. 




D-2. Evaluator char- 
acteristics (selected) 

^(commitment to use; 
political sensitivity) 



1. 
2. 
3. 
4. 
5. 



Organizing For Evaluation Use 

;:)ifoirkS§TS3et O(Continued) 



/ 



Factor Affecting Use: 



Relevant Information: 



Things I Can Do To Influence Use: 



D-3. Information 
dialogue-— summative 
(amount, type and quality 
of interaction between 
evaluator and user(s)) 



1. 
2: 
3. 
4. 
5. 



1. 
2. 
3. 
4. 
5. 



'A- 



Dr4. User commitment 
to use 



3. 
4. 



1. 
2. 
3. 
4. 
5. 
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